Performance & Evaluation

Evaluation Metrics and Criteria

Your submissions will be evaluated based on a score combining accuracy and bias (both normalized as percentages). To ensure a fair comparison, we censor shortages: the score is only computed during weeks with at least four days of available inventory. 

In practice, the following code will be used to evaluate submissions, 

def data_competition_evaluation(phase="Phase 2", name=""):

   # Submission should be loaded from a .csv file.   

   submission = pd.read_csv(name)

   assert all(col in submission.columns for col in ["Client", "Warehouse", "Product"])

   submission = submission.set_index(["Client", "Warehouse", "Product"])

   submission.columns = pd.to_datetime(submission.columns)

   assert (~submission.isnull().any().any())

   # Load Objective

   objective = pd.read_csv(f"{phase} - Sales.csv").set_index(["Client", "Warehouse", "Product"])

   objective.columns = pd.to_datetime(objective.columns)

   assert (submission.index == objective.index).all()

   assert (submission.columns == objective.columns).all()

   # Remove cases where we have 3 days of stock or less

   # This is an important rule that we communicate to competitors.

   in_stock = pd.read_csv(f"{phase} - Inventory.csv").set_index(["Client", "Warehouse", "Product"])

   in_stock.columns = pd.to_datetime(in_stock.columns)

   in_stock = in_stock.fillna(7) #If na we assume it's in stock.

   in_stock = in_stock > 3 

   objective[~in_stock] = np.nan

   # This is an important rule that we communicate to competitors.

   abs_err = np.nansum(abs(submission - objective))

   err = np.nansum((submission - objective))

   score = abs_err + abs(err)

    score /= objective.sum().sum()

   print(f"{name}:,", score) #It's a percentage

Benchmark or Baseline

Participants can use provided baselines for comparison. They serve as a starting point for participants to build and refine their models, aiming to surpass these initial benchmarks in terms of accuracy and efficiency.

By adhering to these evaluation metrics and leveraging the provided baselines, participants can demonstrate the robustness and effectiveness of their predictive models in forecasting sales trends effectively.

Presentation and Documentation

To submit a prediction, follow these rules,  

• Submit your forecasts in a .csv file 
• Make sure there are no missing values in your submissions.  
• Submit a forecast for all products and warehouses. 
• You need to forecast 13 weeks of demand.  
• We will censor predictions during weeks with shortages (defined as weeks with four days or more without inventory.)  
• For Phase 2, you must submit your commented code with your forecasts. We need to be able to replicate the winning forecasts; if you use stochastic models, please set a seed to get reproducible results.