Scaling Data Teams

Last updated:
Scaling Data Teams

Original content: Data Teams with Rya Sciban on SE Daily Podcast

Interesting podcast episode on Software Engineering Daily, where the hosts talk about what specific needs data teams have and what changes when you go from small-scale to large-scale data science.

5 Needs of data teams

  • Having easy access to data (preferably at a single location throughout the organization)

  • Having the means to analyze and visualize data (i.e. good tools)

  • Being able to easily reproduce analyses and research (again, good tools but also processes)

  • Sharing insights and findings in the form of reports and or presentations (ditto)

  • Automating recurrent work (maybe built into tools, but often manually scripted)

Data Analysts vs Data Scientists

Data Analysts Data Scientists
Work with SQL, Excel Work with scripting tools such as R, Python or SAS
Focus on reporting, using existing data Focus on training ML models for predictive or exploratory analysis

Relational vs Non-relational Databases

There's a tradeoff between flexibility and consistency.

Non-relational databases allow for more flexibility but there may be a higher risk of inconsistency (because the data may not be normalized as in relational databases)

Row-based vs Column-based data stores

Column-based data stores are more suitable for most analytics workflows because you are very often interested in viewing a single attribute over many records (as opposed to viewing the same record over many attributes)

Row-based stores sometimes allow you do remodel your existing row-based data as columns, for example using the PostgreSQL Columnar Store Extension for Postgres versions starting from 9.3.

io-utilization-for-postgres-citusdata-columnar-store Dramatically lower IO usage when using columnar data store (brown
and black) vs regular Postgres data types (blue)

Excel is still useful at many scales

It's easy to use and intuitive.

Especially for exploratory and ad-hoc (one-off) analysis.

Stakeholder communication

Try to understand the user's actual problem and needs rather than just blindly implementing what you're asked to do.

By talking (really listening) to your stakeholders you can help them understand the data work your team is doing you can suggest other ways of tackling problems.

For example, developing a simpler solution that covers 80% of their needs for 20% of the cost is a massive win.


If you build data products, you want to collect metrics so that you know what specific features your users/stakeholders are using, for example:

  • What features are more often used, by what types of users?

  • What features are really not worth their screen real estate?

    • I.e. maybe you could hide them from the product's UI so as to reduce clutter and improve usability?

This short post is part of the Data Newsletter. Click here to sign up.


Dialogue & Discussion