Collaboration in an unstructured world

While I haven’t made any decisions yet, I’m currently considering heading off into a much less structured career path which could be variously described as “independent scientist”, “entrepreneuer”, “contractor” or “self-employed open-source developer”. Along those lines, I’ve been wondering what makes work good for me and how I can put the pieces together reliably. I think the critical piece will be to proactively communicate with lots of people.

At the moment, I work as a machine learning engineer for QuantCo, working together with business people, economists, statisticians and other software engineers to build useful software systems for things like ecommerce pricing or insurance fraud detection. Jobs like this are inherently collaborative. Half the job is to work together with other people to figure out what exactly you should be doing and how to do it. There are strategy discussions, design discussions, pair programming sessions, stand ups and just plan old hang-out-and-chats.

By comparison, as an independent scientist, there’s not necessarily anyone there that is depending on me, helping me or working together with me. Even as a faculty member within the structure of academia, there’s a lot of individualism to science. A single research group is often quite disconnected from the rest of the community. Papers aren’t normally communicated until they’re done. A lot of people simply don’t care about the work you’re doing. Seminars or conferences can be confrontational. People don’t often say what they really think about some research outside of their small social network.

So, as I’ve been considering this potential path, I’ve been thinking a lot about how to establish the kind of collaboration I want. And I think part of the solution is to really focus on the people. While the true goal of my work is scientific progress combined with personal fulfullment, a reasonable proxy goal might be to make it all about people and engagement: people that value/use my work, people that talk to me, people that work with me. I doubt the end result will be very different between these objective functions and the people-focused version is going to do a better job of maintaining my momentum.

Along those lines, these are some rough ideas I’ve had:

• Find one or more close collaborators. This is my number one collaborative goal – to find someone who is interested in working together on a daily basis. Not your typical weekly arms-length collaboration. I don’t know if this goal is achievable.
• Do projects for people. Facilitate other people’s work as much as I can. Make it undeniably valuable for other people to seek me out and work with me.
• Write a lot. I probably won’t be publishing articles in journals, but even if I were, the 6-12 month publishing cadence is far too slow to rely completely on publications for communicating with other scientists. I’d prefer a feedback loop on the order of a week or two. I plan to use my website for my technical writing.
• Email people on a regular basis. Cold emailing people. Build an email list that I use any time I write a new website post. Recruit hard for my email list. Should I become the guy-who-spams-his-website-posts-to-everyone? Is this a case of no publicity is bad publicity? I’m not sure.
• Lots of video calls! Visit in person once the pandemic is over if people are in the northeast.

Three pillars of good work

While discussing some career ideas with a friend recently, she asked me to back up and think about what basic components make work good and productive for me. Below was my attempt at a response:

1. A medium collaboration level. Maybe ~10-40% of time is spent communicating and working together with good people who are invested in the work I’m doing and I’m invested in their work.
2. Work that is intrinsically motivating. This is a fairly broad category. That could be building something that I think people want, could be fixing something that is broken, could be learning something new.
3. Enough control. I don’t need to be master of the universe, but without some control, people will treat you poorly, projects will get cancelled and any number of other bad things will happen.

I think I’ve had two of these things at many points and that was super motivating and resulted in high happiness and high productivity:

• During grad school, early on, I had instrinsically motivating work (#2) and plenty of control (#3). That was really awesome. I lost some of both due to some of the mistakes I made early on and getting stuck by the need to finish my dissertation when I also knew I was sort of working on a dead-end research-wise that very few people would be interested in. In the absence of organizational requirements like submitting a dissertation, I would’ve made a huge pivot in direction around year 4 or so. Maybe I should have made that pivot anyway, but I’m not sure. More collaboration (#1) would’ve have helped a lot in a couple critical points to help me avoid the mistakes I made.
• I also had #2 and #3 when I was in high school playing with deep learning. Yeah, deep learning! Way back in 2007! I was super motivated and implemented some cool autoencoder stuff. I just wasn’t enough of an adult to continue down that path. I needed to have a lot of collaboration with people with more experience and I had none. I needed several years more math skills. I needed to go get in touch with some faculty or grad students working on this stuff. So, this project fell short of its potential due to a lack of collaboration (#1).
• While working at QuantCo, at various points, I’ve had #1 and #2. Last year, I was working on improving a forecasting machine learning system at client-name-removed and that definitely qualified as #2 and I had enough interaction with other team members and client stakeholders that I definitely had #1. It was pretty great and I was super motivated. The project was shut down so this ended because I didn’t have control (#3).
• I had #1 and #2 when I worked as a software engineer for a web-based therapy records system called TherapyCharts the summer after high school. Fast, dynamic work in an office with the founders. I continued to work on TherapyCharts during college on and off for a couple years as a part-time contractor. But, I had much less motivation. It had become clear to me that the company was not going to be successful and I felt like I needed to jump ship. But, maybe more important was that it had become much less collaborative because I was several hundred miles away and I was only communicating with the founders every few days or once a week.

I took away two lessons from this list. First, when I’ve felt unfocused or unable to get good work done, it was because something was fundamentally wrong and not because I needed to push harder. You can’t push through a fundamentally flawed mathematical method or a flawed business plan. When things went from going great to going badly in the situations above, I probably should’ve made changes instead of just languishing or pushing against a brick wall. Second, any time I can think of where I had two of these things was a really good time for my motivation and happiness. I don’t know what would happen if I had all three of these things and maintained that situation for a few years. Probably something really good. I should probably try to find that!

Thanks to Liz Santorella, Ruth Byers, Brendan Meade, Reena Joubert, and Phoebe DeVries for helping me crystallize some of the ideas in this post and for providing some edits. The ideas still are pretty rough and hypothetical, so if you want to chat please get in touch!