Lujing has lived everywhere from SoCal to Toronto to Nanjing to Sugar Land, Texas (we couldn't fit the rest)! A security enthusiast, he organized his high school's first CTF. He also firmly believes in the motto: "Food is life".
Lujing joined the infrastructure team this summer and worked on a number of the team's critical projects. After starting his summer by building extensible support for user and role permissioning in our system (and patching some security bugs in the process), he built a plan limitation enforcement mechanism into our server. He also finished building a translation layer between our timeseries DB, Druid and GraphQL and adding that to our GraphQL API, with some nifty optimizations.
Interview
How did you get interested in computer science?
I was bored in school and I played a lot of video games. Then I stumbled on trial software. You know the ones that let you use this full version for 14 days, so I looked into how to get around these particular limits. That's how I got into the security side of computer science. I did that for quite a while and then in high school freshman year I was introduced to python and my science teacher introduced me to this site, a security competion called csaw. I had a lot of fun and we placed first in my sophomore year. That's how I really got into computer science.
First it was the computer security portion and then I started to enjoy building software for challenging problems that are interesting to solve. What I've started to really like is developer tools and making an impact through education. In junior high school, we ran own capture the flag competition. That was the first time I built a complex system that handled the scoreboard and designed challenges. It was really fun. We begged a few companies for sponsorship and got help from CMU. Around 100 competitors participated. With that process, I realized I can make impactful things that teach other people. With my passion about computer science I love seeing the impact it can make on other people.
How have you kept pursuing your passion since continuing college?
During 6.009, I help people find bugs 9 hours a week and really like helping people discover more stuff in computer science. It's super cool and feels really awesome. I also build my own side-projects. When I have time I browse the the ctf community, but I really don't play competitively anymore, since it's a huge time investment
Lately, I've been focussing on algorithms and theory classes because have had practical experience. Right now I'm definitely learning a lot and exploring different areas of cs at MIT
You've done a lot over the summer, can you talk a little about it?
I started with permissioning, then lazily update schemas when out of date, built a portion of the druid query layer, and finished by adding dataloader to optimize our GraphQL gateway.
The gateway portion was a really interesting problem. I pulled up Engine one day and was seeing 1.5 to 2s load times. I was like what's a P95 on API Gateway and the I listed our most used operations. Turns out there was a query that was taking four times longer than every other query. I realize that it was actually the main page query that lists services for all time.
Next I tried to figure out what was actually slow in the query and so I went to look at the traces tab in engine. The p95 was slow because gcs fetching was slow. I realized that we always fetch schema from gcs every time. This is unnecessary since if someone retrieves it, they're probably going to access it again. Then I thought maybe we can keep it in memory somehow, so I installed an LRU cache.
The other thing I noticed that databases were timing out and it was taking 20 seconds to get a connection. I realized that services for an account are fetched one at a time. I realized we could resolve the services with dataloader, batching on the event loop and resolve everything at once. Essentially turning 2n queries into 2 queries.
Over the past few weeks, the p95 is now less than half of what used to be and medium time is down 30% Also the p99 went down by 1/3. In terms of real impact, it's definitely faster for most users. After the summer, I've definitely learned a lot about stability and scaling.
Favorite part that wasn't technical?
Curling was incredible. It was really fun scrubbing the ice. It exercised a lot of muscles I hadn't used before. I really enjoyed seeing everyone really excited. Saling was also nice, except I put sunscreen on everywhere except my face. The food at the sailing event definitely made up for it, though.
This was a really collaborative summer, can you tell me more about that?
During the beginning of the summer, I worked with Martin on setting up a development stack. Towards the end of July, I started to work with the frontend team and building up gateway, the server that sits between the client request and our Kotlin GraphQL server. The frontend was putting out a ton of new features, like schema diff and versioning features. I was excited to work with frontend to design a schema and determine how operations should work. It was cool to see how graphql facilitates communication just as a common interface. I really liked collaborating with the other teams and interns. It was a great chance to see my work being used by other people.
Coming in knowing nothing about GraphQL, you're now an expert. Where do you see it going?
My first thought coming in was actually that people don't need GraphQL, since it puts more work on the backend developer. I realized that it's not just for replacing a REST api. GraphQL also allows you to manage data in a consistent graph. Apollo has built up frameworks to make it easy. Working with Apollo, it does magic.
I see that anyone who has REST endpoints will want to switch to GraphQL. Right now open source is super important and provides foothold in community. For me being a core part of the graphql community is really cool. I think a lot of people really respect how knowledgeable we are about GraphQL in general. On top of that we're getting a ton of big customers.
You wanted to learn about systems, how well do you think you accomplished that goal?
It's been amazing how much I learned about systems in general. I came in with cursory knowledge of databases and docker containers. I also knew Java fairly well. As time went on, I learned more about how parts interact and about distributed systems. I now understand why Druid as a database in java could be 100x faster than mySQL which is in C. It's really cool learning about how to to horizontally scale things and make things resilient and tolerant to failure. How to monitor things has been great to learn, too and knowing when things are broken is super important to me now.
After learning a lot of distributed systems and performance engineering, it's encouraging to pursue more classes in distributed systems and databases. It also got me excited to take my next algorithms class after I saw a problem set solution in Druid's group by query. It's finally nice to see systems engineering knowledge is actually very useful at work.
What advice would you have for people joining Apollo?
Come with an open mind, especially be receptive to all the new stuff and be ready for the rapid pace. Also really embrace the collaborative culture here, which I think is really awesome. Stay open to learning a ton of new stuff. It's really quite cool.