Having worked in the industry for 6 years now, I’ve seen machine learning projects succeed but, most importantly, I’ve seen many more fail - I even failed some of them myself due to inexperience and/or poor judgement. Although each failure has its own story, reasons and learnings, some common denominators always exist. One such important denominator can be “boiled down” to a single assertion that - as an ML engineer - I am particularly fond of:...
the problem A common workflow in machine learning projects (especially in Kaggle competitions) is: train your ML model in a training dataset. tune and validate your ML model in a validation dataset (typically is a discrete fraction of the training dataset). finally, assess the actual generalization ability of your ML model in a “held-out” test dataset. This strategy is widely accepted, as it forces the practitioner to interact with the ever important test dataset only once, at the end of the model selection process - and purely for performance assessment purposes....