VirusTotal with Emi Martínez

MELANIE: I'm good. How are you, Mark?

MARK: I'm doing well.

MELANIE: Great.

MARK: I'm not what's happening today. I feel like I have a very deep voice today. [LAUGHS]

(DEEP VOICE) I'm doing well. [LAUGHS]

MELANIE: You do. (DEEP VOICE) You have a very deep voice. I'm your father.

(NORMAL VOICE) OK, so Mark.

MARK: Yeah.

MELANIE: Tell me about this week.

MARK: Yeah, this week, we're going to chat with Emi Martínez, who is the technical lead for a company called VirusTotal. VirusTotal is a company that does a lot of really interesting stuff in the security space.

MELANIE: Yes, we like security. Well, as always, before we get into the interview, we will start off with our cool things of the week. And we will end with the question of the week. And this week's question, we have a special guest who's going to help answer a question that we got from one of our listeners, the question being that "what is the best practice for multi-tenancy in Google Cloud Spanner, especially if customers are not of the same size and have unequal load?" And that's coming from Andrew Sheridan. So yeah, we'll have Robert Kubis a little bit later.

All right, to cool things of the week. So one of the cool things is that, basically, this last month, they've added a new data type, new data import formats, and finer cost controls. And so the new data type is numeric. And the data formats for ingest are Parquet and ORC. And so that's basically available as a natively supported format for ingest into BigQuery.

And then, like I said, there'll be some more finer cost controls. And that's all on the blog post that we will include in the show notes.

Another cool thing of the week is that Dataflow Stream processing now supports Python, which is a great thing to be supporting because, frankly, Python is such an important language in the data space. So now you can use that with Dataflow, Dataflow stream, I should say.

MARK: My Python is terrible, like, just really bad.

MELANIE: But yeah, that's fine because Python loves everyone. Python will be very easy for you if you ever want to approach it.

MARK: I've tried a couple of times.

[OMINOUS MUSIC, SNAKE CHATTERING]

SPEAKER: (BRITISH ACCENT) Mark realizes that his approach has enraged the Python, and to be safe, he must retreat.

MELANIE: Oh, you did?

MARK: It's interesting. Yeah, I did some Addspell stuff in Python.

MELANIE: What were you doing wrong?

MARK: [LAUGHS] I don't know.

MELANIE: Anyways.

MARK: One other cool thing of the week is that we're announcing a new certification for Google Cloud Certified, the Associate Cloud Engineer.

MELANIE: Yes, and this is in combination with some of the other certifications that we have out there, like Professional Cloud Architect, which came out in 2016, and also Data Engineer. So this is just another way to further expand and become an expert in the space.

MARK: Yeah, so if you're interested, an Associate Cloud Engineer is someone that deploys applications, monitors operations, and manages enterprise solutions.

MELANIE: There you go.

MARK: So if that's something that you think you fit into, this certification may well be for you.

MELANIE: Last cool thing of the week that we wanted to mention is relation to Next, which many of you have heard us talk about before, is coming up later this month.

MARK: Oh, my god. I better get some stuff done.

MELANIE: Yeah, you got lot of work to do.

MARK: I'd better do some stuff.

MELANIE: I think we both do. So there is a blog post that's out that emphasizes some of the AI, machine learning, talks that are going to be going on that people should to check out, some of them being like how to get started injecting AI into your applications, architecting live in CAA predictions from archives to insights. This was something we mentioned in cool things a couple months back. Eric Schmidt, one of our colleagues, was working on this.

And we also have another colleague, Martin Gorner, who has been doing this whole series around TensorFlow, deep learning, and modern convolutional neural nets without a PhD. So all of these things will be at Next.

MARK: Yeah, it's actually worth noting, if you haven't been back to the Next website for a while, there's actual times for all the sessions. So you can actually go in and see what everything is doing, all the tracks, and everything that's going on. So there is a lot of content.

MELANIE: That's great.

MARK: Yeah, so definitely go check it out. If you register, if you have your tickets, go in, sign up, make sure you have a bit of an idea. I think it's going to take some planning. There is really a lot of content.

MELANIE: That sounds fun.

MARK: Awesome. Well, why don't we go chat with our friend Emi over VirusTotal and talk about all the security things.

MELANIE: Sounds good.

MARK: Let's do that. So we have a really interesting project today. Emi Martínez is here, VirusTotal tech lead, to talk to us about VirusTotal today. Emi, how you doing today?

EMI: I'm doing great. Thank you.

MARK: Thank you so much for coming to join us. VirusTotal is super interesting. But before we start talking about that, do you want to tell us a little about you and what you do here at Google?

SPEAKER: OK, so I'm actually tech lead for VirusTotal. And that's no more that I am a software engineer that gets to work with legal, with marketing, with sales, et cetera. I happen to actually bootstrap the sales at VT at the time, because after acquisition, we didn't have any sales personnel, and before that, we didn't have either. So basically, I was wearing the engineering hat in the mornings, and in the afternoon, I was doing sales. So what it means is I have a broad visibility of the product, and I can answer many questions.

MARK: Excellent. Well, OK, so at a high level, what is VirusTotal?

EMI: Yeah, so most people think that VirusTotal is simply a public-facing website to which you can upload a file and get the verdicts from more than 70 anti-virus solutions in order to get a second opinion about the maliciousness of a file. And that's the basic understanding for the novel users.

If you are a bit more curious, you probably have discovered that we not only scan the files, but also URLs, domains, IP addresses. And the bottom line is that we're receiving about 2 million files a day. And we're doing tons of URLs, scanning, and domains, and IPs.

So eventually, what happens is, we are not only running AVs on files. We are also running all sorts of tools that extract metadata and relationships about files and the other items that we scan. So for instance, when you upload that file, and it happens to be an executable, we will run it in a ritualized environment. And we'll get to see what that file does when it infects a machine. We will then run all sorts of micro-services that, for instance, if you upload an executable file to VirusTotal, it will try to accept file to VirusTotal at any compressed bundle, but will try to decompress it, and if there's any interesting files within that bundle, re-upload to VirusTotal and get the inner files scanned. So we do tons of micro-services on the files' URLs and the rest of items that we scan.

And eventually, what happens with all this data is it gets into our dataset, which these days, is over 2 billion files. And the end product of all this work is we get to create a service, which I usually describe as Google and Facebook, you mix them together, and you apply them to the malware field.

So why Google? Because we created this thing called VirusTotal Intelligence, which is like a search engine over malware. And you get to use over 40 modifiers in order to identify particular malware that interests you. So for instance, you can do stuff like, hey, let me know all of the malware that reaches out to Google Drive upon execution, or all the malware that is detected by a given anti-virus engine with a given label, and happens to be a PDF that contains JavaScript. So you can combine more than 40 modifiers in order to identify particular families of malware that interest you.

So that's the Google side of things. Why do I say that's similar to Facebook? Similar to Facebook in that for each match of your search results, you get to see a profile of every sample in the dataset. So you get to understand the individual items of that malware and the individual behavior, thanks to the sandboxing and virtual environment execution that we do.

But you also get to see the relationships with other items in the dataset. So has this file been downloaded from a given URL? Or is this file reaching out to a given domain? And what other files are reaching out to those resources? So it builds in this relationship notion that is similar to the way that Facebook works, but applied to malware.

So we build this intelligence tool of malware that many security companies, security teams worldwide use. And it has its counterpart as an API, the same functionality, but exposed programmatically so that you can integrate with your programs with the code dataset that VirusTotal built.

And that's like a very broad overview of what we do. [LAUGHS]

MARK: Is this something we're doing in isolation? Or are we doing this in conjunction with other security researchers or anything like that?

EMI: Yeah, so the thing, the technology itself, we build it here in VirusTotal. But it's used by thousands of researchers worldwide. So the first people that use it are the anti-virus vendors participating in VirusTotal, which are over 70 of them, including companies like Simon Tech, Kaspersky, McAfee, et cetera.

And the second set of users that will be using this is security teams in big and medium companies. So at the end of the day, they use the tool and generate knowledge about malware that then contributes to the worldwide defenses against the bad guys.

So in some way, we are an ecosystem where all the security-related individuals co-exist and work with each other, sharing the knowledge through VirusTotal.

MELANIE: How did the VirusTotal idea get off the ground to begin with?

EMI: So it's a very interesting story. The founder, Bernardo, way back, early 2000, he was doing anti-virus comparatives, so performance comparatives of anti-viruses for a magazine. And he had two needs. One, he needed to compare these anti-virus solutions. And doing it manually was difficult. So he wanted a way to automate that task. And secondly, he needed samples, malware samples, in order to test this product. OK?

So he hired a team that went on to build the very first VirusTotal. And since he had this need, and he had already built this, he said, hey, why not expose it publicly because I'm going to receive malware samples. I'm going to give value to other users that might have a similar use case.

So these days, we don't promote at all anti-virus comparatives with VT. It's not something you should be doing. But that's how it was born.

And thereafter, it's grown into a tool that mostly any security researcher in the world is using in order to have a second opinion about files. And since we are receiving 2 million files per day--

MARK: Wow.

EMI: --we get to have a massive dataset with all the behaviors, with all the related items. And hence, these days it's more of a malware research tool rather than its initial use case.

MARK: When people submit stuff, is this an entirely automated process? Do we have security researchers on our side also doing stuff? Is it combined? What's that back end process look like?

EMI: OK. So what happens is, either they use the API or the public web interface, which, under the hood, uses the API. So you submit a file. This reaches certain Google Cloud infrastructure, which happens to be in App Engine. And App Engine is serving both the public facing front, and it's also acting as a back end for the Datastore.

When the file reaches App Engine, we copy it to Google Storage and create a task to say, hey, analyze this. And this task goes into Pub/Sub systems and micro-services that we run at our end. For instance, one of them is simply all of the AVs running into Google Compute Engine. So we'll subscribe to this Pub/Sub up topic, get the file, scan it with all the anti-viruses, send back the results, either via Pub/Sub, or an HTTP post in an internal API to App Engine and store it in the Datastore.

Thereafter, once we have all this knowledge in the dataset, that's when all the researchers come over, because either they are checking hashes and files against VT, and they happen to comment on those files, and investigate through the web interface, or because they have some kind of automation, pulling our feeds of malware et cetera. So that's the way it works.

And the end output of this is that what they research with VT they end up creating rows to detect it. And so this, in some sense, nurtures the entire security industry. And we're making a dent on billions of users through this security product that happen to do their research with VT-- all the AVs using VT, security teams in big companies that try to secure their users, et cetera.

MARK: That's super cool. I'm curious to know, is it like-- I mean, just on the GCP side of things, why did you choose App Engine? Has that been a really great fit for you? How's that working?

EMI: So if I rewind back in time before the Google acquisition, we actually run our own data center in our office. And it was super noisy. And you didn't want to be there. You know?

And what happened is, we had a couple of MySQLs. And we have had a couple of front ends in Apache. And we end up spending more time managing the systems and doing administration rather than coding. It was horrible. Every single day, we would have some kind of MySQL corruption that will lead to downtime.

So once upon a time, before the acquisition, Google came over and actually offered us a deal to go on to App Engine for free in exchange for malware intelligence. And that's how our initial relationship got forged. And we started building in App Engine.

And I wouldn't ever return to the initial days. Right now, we can focus exclusively on coding, rather than on managing systems. And for us, it's really seamless because it automatically scales. We don't have to worry about scale.

And so it's the best decision we ever made. And probably we wouldn't have been able to grow to the size that we are today, 2 billion files in the collection, without the Google infrastructure. This accounts for two petabytes worth of data in Google storage, you know. And that's only the files they wrote by themselves. Then you have to count all the metadata that we generate for them.

MELANIE: Is any of that data available for people to do research on outside, who are doing research into security issues?

EMI: Yeah, absolutely. We have several offerings. The ones that I mentioned before, where you still do intelligence on our API, they are licensed by third parties. And it's not only internal research in VirusTotal and Google. But rather, if you think about the biggest companies on Earth, they probably are all VirusTotal customers. So are all the anti-virus vendor and security companies worldwide.

So they basically research advanced persistent threats, which are like state-sponsored attacks against their companies. Or they try to address mass malware, which is targeting their users, and stuff like that.

And this ends up, as I said before, in rules to detect malware. And we get to impact billions of users that way, rather than ask directly having billions of users us our users.

MARK: Nice. You mentioned also something else interesting. In the process of things getting processed, you actually grab the file on Compute Engine, download it, and then apply the virus checkers and various other automated systems. Do you isolate this file? I mean, obviously, this might be a file that may contain malware. So how do you make sure that that doesn't spread to potentially other parts of your system?

EMI: When we run the AVs against the files, we don't really execute them. It's like a static analysis, what the AVs do. So in that sense, there could be room for infection if the AV was exploited. You know? But it's not like we are actively infecting the machine.

But if we were exploited, the nice thing is that, since we run this in Compute Engine, we have a snapshot to which we can revert, which would be clean. So that allows us to go back really quickly.

The second thing that we did do that I mentioned is that we execute malware in virtual environments. So this we do, for instance, in stuff like VirtualBox and with some virtualization layer. And thanks to the fact that recently, Google Cloud introduced nested virtualization, we're able to run stuff like VirtualBox inside the Compute Engine also. So that's the way that it's isolated.

MARK: I didn't actually know you could do that. And now, I need to do that. [LAUGHS] I need to run VirtualBox on GCE. That's actually really cool.

EMI: Yeah. We have also a hybrid infrastructure because it's not something that you could do before. But recently, they did this nested virtualization stuff.

MELANIE: Nice. What are some of the more interesting experiences you've had with viruses, or more interesting viruses you've seen that you have done things that were a little out of the norm?

EMI: I would say probably the most interesting thing I can speak of that was discovered, or tracked down and completely understood with VirusTotal is Stuxnet. So if you recall, Stuxnet has been attributed, though not confirmed, to be some kind of malware that the Israeli and US government did in order to slow down the nuclear enrichment program of Iran. So basically, it was a malware targeting the nuclear stations in Iran in order to slow down the centrifuge of uranium in order to enrich uranium. You know?

So the cool thing about this is that it was actually discovered in 2010. And the actual sample was in VT since 2007. So for three years, it was in VT, and no one really knew about it. So when it was discovered-- and I think it was people from Symantec-- they were able to build the entire picture just pulling information from VT, the initial Datastore, where when it was seeing any other samples that were related to that initial attack and so on. So that's probably one of the most interesting ones.

But there's tons, like last year or the year before that, we had WannaCry that was all over the press. And the researcher that discovered the kill switch in order to stop the propagation of that worm was using VT for that.

MARK: Oh, wow.

EMI: Yeah. So it's happening all the time.

Another interesting use case for VT is banking trojans. So you have this strain of trojans that try to target users of electronic banking and others to steal credentials and do automated transactions of money, like wire transfers to the account of either a money mule or the end attacker. And teams within banks and other security companies study these banking trojans all the time with VT in order to understand, for instance, where is the stolen data being sent to, like, which is the command and control for the malware, so that they can take down that command control. And all of a sudden, they might fix tons of infections worldwide, just by taking down the network infrastructure of the trojan.

MARK: Something I also want to highlight, I think, that's actually super interesting, you also run basically what looks like a research community. There's the VirusTotal community. Can you tell us a little bit more about that?

EMI: Yeah. It's over 600,000 researchers that sign up into VirusTotal. There's several reasons why they sign up. One of them is because they want to vote on the files and URLs that the scan with VT in order to, for instance, highlight malicious stuff that's undetected by the anti-virus and security industry today. So that way, they can share this information with the actual malware researchers doing the detections.

Another reason for signing up is that they get to get an API key to do some programmatic work with the VirusTotal dataset. And not only can they vote on items, like I said, they can also place comments, giving you the entire background on the file that they upload. So for instance, on a given file that they uploaded, they might place a comment that has the actual spam email from which the file came. And then there's this really nice contextual information for the antivirus and security industry, because not only they get to see the bullet, but they also get to see what the gun was, and perhaps even they get to understand who the guy with the gun was.

So for many, the end goal is to actually understand and know your enemy. So the bullet is not enough. You have to see the entire picture. And actually, the community helps in that sense.

MELANIE: Where are areas that you'd like the community to provide more help, or more support, or more insights?

EMI: OK. So right now, we are actually doing a nice effort. And we have today the verdicts of of our AV partners. And we recently launched an initiative, which we call the Multisandbox Project. So I mentioned that we have our own virtual set-up where we run samples. Right now, we are collaborating also with some partners and e-companies that want to give us sandboxing execution behavior data for the samples that we have and that they see, so that not only we have the notion of what a file does upon execution from the VT point of view, but also from other points of view.

And this is really important because often, Malware does some kind of cloaking whereby perhaps it won't execute in Spain, whereas it will execute in China. And so if we have partners all over the world, we get to see different behaviors, and we get stealthier against malware that's trying to evade these sandbox systems.

So that's something that I would like to, actually, have more partners working on. Right now, we have, I think, four different companies providing sandbox reports. And we are quickly growing this. But I would like more information in that sense.

MELANIE: How do they get involved?

EMI: It actually started off at something called the Anti-Malware Testing Standards Organization in 2017 or 2018, I think. So one of the AV partners approached us and said, hey, we not only want to be an AV verdicts kind of partner on VT, we would like to also provide our sandbox reports. Why? Because our off-the-shelf solution not only does static detection, but also does behavioral detection. And we want users to know that we have different layers built in to our solution. And if a malware evades the static detection layer, it might get caught by a subsequent layer, for instance, the behavior detection one.

So they approached us and suggested this. It made a lot of sense for us. So we got them on board. And when we did that, we launched this initiative called the Multisandbox Project in our blog. We reached out to our partners to tell them that we were starting this.

And thereafter, VirusTotal, even if it's not known by my mom at home, it's really known in the security industry. So it's just people e-mailing us and saying, hey, we do have a sandbox set up. Can we join?

MARK: So I double checked our website. Apparently, we're clean. I checked gcppodcast.com. And I just wanted to make sure.

MELANIE: That's good to know.

MARK: It's good to know, yeah.

MELANIE: But I'm sure it's also now just made people go, hmm, I wonder if I should try to attack that.

[DYING COMPUTER SOUND]

MARK: Please don't.

MELANIE: Are you hiring?

EMI: So yes, we are hiring. So if you know the story behind us, OK, we were an independent company. We were acquired by Google in 2012. We were integrated in the Google Security team. At one point, it made sense for us to do something bigger, so we went on to Google X. In Google X, there's been a much bigger effort around security, which now has graduated from Google X into an independent Alphabet company called Chronicle.

And Chronicle as a whole is hiring. Chronicle should be the Alphabet company that will address and do the big moonshots in the security landscape. So it's really the place you want to be at. And we are hiring both at VirusTotal in Málaga, Spain, but most importantly, here in the US in Chronicle.

MARK: Fantastic. Before we wrap up today, is there anything else you want to plug, any events you're going to be at, or anything we haven't managed to cover in the podcast to make sure we get everything in.

EMI: [LAUGHS] We do every so often these data connectors events. We are going to be in San Jose on July 12, in Raleigh on July 26, and BSides Las Vegas. And that's an important one because Black Hat, which is the biggest hacker conference, is happening at that time. We will be in August.

So basically, if you want to reach out and meet us, just come by and perhaps send out an email to info@virustotal.com, and we will be very happy to meet you at Black Hat.

MARK: So Emi, is there anything else inside Google Cloud that you use? I mean, you mentioned App Engine and GCE. But you've got lots of data. What are you doing there?

EMI: We are probably the guys that use most distributed knowledges from the user base. I won't bet on that. But for sure, we use a wide variety of tools.

So for instance, I mentioned that every single compressed file that gets uploaded to VT, we decompress, and re-upload the inner contents to VT to see if there is anything malicious there, and to build the relationship, the parent-child relationship. We do that with Google Kubernetes Engine as a microservice that subscribes to the Pub/Sub topic that I mentioned before. So many of the microservices that we run are actually on containers in Google Cloud.

Something else that we can do, for instance, is we are really heavy users of BigQuery. We do it for two things. In order to do analytics over our dataset, and we plug it into stuff like Google Cloud Data Studio to in order to do nice charts and graphs, and understand how users are using the platform.

But also, we mainly use it for advanced malware research and use cases. So for instance, BigQuery has this thing which allows you to insert user defined functions, which you write JavaScript. And it gets executed against every single row that you have in BigQuery. So in the malware field, there's this thing called the ssdeep, which is like a fuzzy hash that allows you to understand similarity between files.

So there's no way you can search for similar files over a 2 billion collection with standard tools. Why? Because it means that you have to compare this hash and the computation against every other item in the dataset. So that's 2 billion comparisons. And you want quick results.

So we plug this into BigQuery. And within seconds, we know what are the files similar to a given sample that interests you. And this is done because we have added distance rather than running as user defined functions in BigQuery.

MARK: Nice.

EMI: For instance, since we are users of the standard App Engine environment but also the Flex environment, whenever we would have to run, for instance, stuff that won't be only Python only, or golang only, so we have containers deployed in Google App and in Flex. And the Flex happens to not have this time that environment memcached. So we are also users of Memorystore, which is a recent thing that was announced. And we're using Memcache with Memorystore and App Engine Flex.

We are also users, for instance, of Google Cloud SQL. Our main dataset is actually in the Cloud Datastore. But we have minor App Engine services which run, for instance, in WordPress for our learning space, and which is VirusTotal.com/learn. So that's our WordPress running in Google App Engine standard environment in the PHP environment. And that connects on SQL, which happens to be a Google Cloud SQL.

So we are also using-- and as I said that we're also using App Engine Flex, not only the standard the environment. We do so when we have a need to run certain code that's not only and Python only or golang only. And since App Engine Flex is a bit different to the standard environment, it doesn't have the Memcache API. So we needed something like Memcache.

So what we are using instead of that as is the Cloud Memorystore offering that recently got launched, which is built with the Redis. And that's been super useful because before that, either we had to call out some kind of internal APIs that were built in the App Engine standard environment. Now, we no longer have to do that.

We are also power users of stuff like the IM administration for our users. So we need to make sure that, for instance, an intern doesn't get to deploy code to our main production site and stuff like that. So we manage that with groups within the G Suite, because VirusTotal.com happens to be also a G Suite account. So we have groups set up there. And then those groups have certain privileges in Google Cloud Dashboard. So that's the way that we manage our users and make sure that no one should be doing what they are not allowed to do.

MARK: That's a lot of stuff.

MELANIE: I know. It's like everything, all the things.

MARK: Oh, and the one fun question I would ask as well-- what languages do you write in?

EMI: Yeah. So most of our code is in Python. We love Python. And it's very typical in the security industry. But we also have some apps and services running with Go and the Go environment. And we also have-- since we have WordPress set up for our very small use case, that's on the PHP runtime.

MELANIE: Great.

MARK: Fantastic. Well, Emi, thank you so much for hanging out with us today. It was super interesting to hear about VirusTotal and what you're all doing in the security space.

EMI: Thank you very much.

MELANIE: Thank you.

MARK: So I want to say thanks to Emi Martínez for joining us today from VirusTotal. That was a super interesting conversation. Security research is really fascinating.

MELANIE: Definitely. I agree with you on that, a very important topic.

MARK: Yeah, absolutely.

MELANIE: OK, well, Mark, question of the week.

MARK: Question of the week.

MELANIE: And this week's questions coming from one of our listeners.

MARK: It is. It's coming from Andy Sheridan, at @andyjsh. He asks, "What is the best practice for multi-tenancy in Google Cloud Spanner, especially if customers are not the same size and potentially have unequal load?"

MELANIE: Right. And so for this week's answer, we actually have one of our colleagues, Robert Kubis, who is joining us to basically answer the question.

MARK: Yeah, thanks for joining us, Robert, because I have no idea. How are you doing today, Robert?

ROBERT: Very good. Thank you for having me. And hello, everyone.

So this is actually a really, really good question. And to basically elaborate a little bit on how Spanner works behind the scenes-- so in Spanner, data is organized based on the primary key. So the primary key you choose, that's how we order the data in Spanner. And then we split these data up in chunks. And these chunk's responsibilities goes to the Spanner notes that you have in your instance.

So now, that means, basically, if you have a multi-tenant application and use a custom ID as an example for a primary key, that all your data, which is belonging to one customer, would go to one node. And basically, this one node is responsible for that.

And that can lead to things like hotspotting. So all your load, which is like all the data that is loaded, all the data that is read, all the queries that are going against this customer, basically have to be handled by one node. Now, you don't want that, especially in cases where you have customers which are of unequal size. So you have some large customers and a lot of small ones.

And a good way to addressing this is to somehow, you have to basically split up your data in a way that the responsibility can be given to multiple nodes in your instance. One of the best practices that a lot of people are doing there is basically prefixing your primary key, or using a larger primary key in your tables. So a prefix could be-- for example, you could use some other element in your table, and you hash that element, and you take that as a prefix in your key. And with that, your data is basically distributed among mutliple nodes. And now, if you have very large customers, it gets distributed between the available resources. And it doesn't matter if you have a lot of large, or like, a couple of large customers and a couple of small ones.

MELANIE: This is great. Robert, thank you so much for joining us today to help answer that question. And if anybody else out there has more questions, please feel free to message us.

MARK: Absolutely. Actually, so usually, when we leave off, we talk about where were going to be. Robert, where are you? Are you doing any events soon? Are you talking about Spanner anywhere if anyone wants to learn more about Spanner, since we have you here?

ROBERT: Yeah, sure. So the next big event that is coming up for us is, of course, Next in San Francisco. I think by now everybody is speaking about it.

I will be there. I have two sessions. And one is actually on advanced topics in Cloud Spanner, like how to design your schemas to really get the best out of Cloud Spanner. So I'm going to talk about this. I'm going to talk a couple about metrics, or how to tune Cloud Spanner, basically, to the best of your needs.

MARK: Awesome.

MELANIE: That's great.

MARK: Fantastic.

MELANIE: Mark, where are you going to be?

MARK: So yeah, I'm just like Robert. I'm going to be at Next. I'm going to speaking about Agones, as the open source project I've been working on for running game servers, and I think maybe doing some other stuff.

But yeah, you and I are going to be there, Melanie, doing some podcasty stuff.

MELANIE: Yes, we will.

MARK: And Melanie, where are you going to be? What are you up to?

MELANIE: I am going to be speaking, actually, in the next couple of weeks at PyCon Russia.

MARK: Oh, cool.

MELANIE: Yeah. I will be speaking about reinforcement learning on the 22nd of July.

MARK: Excellent.

MELANIE: Yeah.

MARK: All right.

MELANIE: But then, I'll be at Next.

MARK: Wonderful.

MELANIE: As will we all.

MARK: Awesome. Well, Melanie, Robert, thank you very much for joining us on another weekly episode of the podcast.

MELANIE: Thank you, Mark.

ROBERT: Thank you for having me.

MARK: All right. And thank you all for listening. And we'll see you all next week.

[MUSIC PLAYING]