Category Archives: Kerika

Posts about Kerika, the company and its people

Why we are integrating with Box; Part 2: Transparency

Kerika, Technology,

(The second in a series of blog posts on why we are adding integration with Box, as an alternative to our old integration with Google Drive.)

In our last post we talked about the privacy overhang that Google Drive faces with many of our prospective customers.

In this post we talk about some frustrations we have experienced as a Google Drive developer, which we would characterize as generally flowing from a lack of transparency on Google’s part.

Sometimes things break within Google Docs or Google Drive, and if the outage is not very widespread, it isn’t acknowledged very well. Google has an App Status Dashboard that is useful when there are major problems, but we have seen minor issues persist for days at a time without being reflected on this Dashboard.

On one occasion there was a problem with Google’s OAuth infrastructure: people trying to sign into their Kerika accounts were getting bounced with mysterious error messages. There was no easy way for us to figure this out because OAuth is simply an underpinning service for Google Apps: it doesn’t have its own status indicator on the App Status Dashboard.

We spent a day sifting through error messages and exceptions logged by our server (in increasing desperation!), before the problem went away as mysteriously as it had appeared.

More recently, our users faced a problem with opening PDFs and image files (PNG, JPG, etc.) that were attached to cards or canvases.

Kerika was correctly getting the thumbnails of these images from Google Docs — so we knew the files were there and were accessible by the user (i.e. it wasn’t a permissions problem), but when users clicked on the thumbnails they got a “503 System Error”.

This particular problem hit close to home for us: we use screenshots and mockups very extensively to communicate between our Seattle and India-based teams.  Not having an easy way to download images that were attached to cards and canvases was a serious inconvenience for us, and we even resorted to using email! (We had given up using email over 18 months ago…)

Being able to seamlessly manage your cloud storage files is a critical element of the Kerika user experience, so from our perspective this was a major problem. Every attempt at debugging failed: old files were opening correctly, but new files weren’t. And while documents and spreadsheets were opening correctly, images and PDFs wouldn’t open…

We even combed through our git branches to see where the bug might be hiding.

There as no bug on our end: our code dealing with thumbnails and Google Drive hadn’t changed in the past 6 weeks.

At this point we knew the problem wasn’t on our end, but that still left us with frustrated users. Our users have very high expectations of Kerika’s product quality and customer service, and we couldn’t explain the problem or manage expectations.

As software professionals ourselves, we are entirely sympathetic to others facing bugs or outages: we suffer these too, all the time, and take no pleasure in casting stones. But we do think Google could have done a better job of notifying people that there was a problem, so we could manage our users’ expectations accordingly.

This is what we try to do ourselves: when we find a bug, we reach out, proactively, to our users to let them know they were affected. We try to provide an honest description of what happened (if it’s a bug in Kerika, we are candid about it), and we provide updates while we work to fix the bug, and then we reach out to them later to make sure our fix works.

Here’s how our users react to our outreach:

“AWESOME! thanks! I’m recommending you to all my friends.” — a church pastor in Florida

“Woo! Yes. Thanks very much!” — marketing manager in Colorado

“WOW!  So impressed with your response time and thoughtfulness.” — small business owner in California

We concluded that any alternative cloud platform we chose had to come from a company that was more transparent and accessible: we wanted to be able to able to report problems and get them fixed, and most importantly, we wanted reliable channels of communication. If we know what’s going on with the cloud platform, it becomes much easier to manage our user’s expectations and keep them happy.

The full series:

 

 

Why we are integrating with Box; Part 2: Transparency

Kerika, Technology,

(The second in a series of blog posts on why we are adding integration with Box, as an alternative to our old integration with Google Drive.)

In our last post we talked about the privacy overhang that Google Drive faces with many of our prospective customers.

In this post we talk about some frustrations we have experienced as a Google Drive developer, which we would characterize as generally flowing from a lack of transparency on Google’s part.

Sometimes things break within Google Docs or Google Drive, and if the outage is not very widespread, it isn’t acknowledged very well. Google has an App Status Dashboard that is useful when there are major problems, but we have seen minor issues persist for days at a time without being reflected on this Dashboard.

On one occasion there was a problem with Google’s OAuth infrastructure: people trying to sign into their Kerika accounts were getting bounced with mysterious error messages. There was no easy way for us to figure this out because OAuth is simply an underpinning service for Google Apps: it doesn’t have its own status indicator on the App Status Dashboard.

We spent a day sifting through error messages and exceptions logged by our server (in increasing desperation!), before the problem went away as mysteriously as it had appeared.

More recently, our users faced a problem with opening PDFs and image files (PNG, JPG, etc.) that were attached to cards or canvases.

Kerika was correctly getting the thumbnails of these images from Google Docs — so we knew the files were there and were accessible by the user (i.e. it wasn’t a permissions problem), but when users clicked on the thumbnails they got a “503 System Error”.

This particular problem hit close to home for us: we use screenshots and mockups very extensively to communicate between our Seattle and India-based teams.  Not having an easy way to download images that were attached to cards and canvases was a serious inconvenience for us, and we even resorted to using email! (We had given up using email over 18 months ago…)

Being able to seamlessly manage your cloud storage files is a critical element of the Kerika user experience, so from our perspective this was a major problem. Every attempt at debugging failed: old files were opening correctly, but new files weren’t. And while documents and spreadsheets were opening correctly, images and PDFs wouldn’t open…

We even combed through our git branches to see where the bug might be hiding.

There as no bug on our end: our code dealing with thumbnails and Google Drive hadn’t changed in the past 6 weeks.

At this point we knew the problem wasn’t on our end, but that still left us with frustrated users. Our users have very high expectations of Kerika’s product quality and customer service, and we couldn’t explain the problem or manage expectations.

As software professionals ourselves, we are entirely sympathetic to others facing bugs or outages: we suffer these too, all the time, and take no pleasure in casting stones. But we do think Google could have done a better job of notifying people that there was a problem, so we could manage our users’ expectations accordingly.

This is what we try to do ourselves: when we find a bug, we reach out, proactively, to our users to let them know they were affected. We try to provide an honest description of what happened (if it’s a bug in Kerika, we are candid about it), and we provide updates while we work to fix the bug, and then we reach out to them later to make sure our fix works.

Here’s how our users react to our outreach:

“AWESOME! thanks! I’m recommending you to all my friends.” — a church pastor in Florida

“Woo! Yes. Thanks very much!” — marketing manager in Colorado

“WOW!  So impressed with your response time and thoughtfulness.” — small business owner in California

We concluded that any alternative cloud platform we chose had to come from a company that was more transparent and accessible: we wanted to be able to able to report problems and get them fixed, and most importantly, we wanted reliable channels of communication. If we know what’s going on with the cloud platform, it becomes much easier to manage our user’s expectations and keep them happy.

The full series:

 

 

Why we are integrating with Box; Part 1: Privacy Overhang

Kerika, Technology,

(The first in a series of blog posts explaining why, and how, we are adding Box as a cloud service in addition to our long-standing integration with Google Drive.)

When we first started working on Kerika, back in 2010, Google Docs was an obvious choice for us to integrate with: it was pretty much the only browser-based office suite available (Microsoft Office 365 wasn’t around and Zoho was, and remains, somewhat obscure), and we were quite sure we didn’t want to get in the business of storing people’s desktop files in the cloud.

Google Docs (not yet renamed Google Drive) did have various limitations in terms of the file types it would support, and further limitations on the maximum size permitted of different types of files — the largest spreadsheet that they would support, for example, wasn’t the same size as the largest Word document — but the idea of building Kerika as a pure-browser app was a very appealing one.

So we integrated with Google Docs at a low level, fully embracing their API, to provide a seamless experience unlike anything that you might find from competing products:

  • When you add a document to a card or canvas in a Kerika board, it is automatically uploaded to your Google Drive, where it is stored in a set of nested folders.
  • When you add people to a Kerika board, their Google Drives automatically get dynamic links to your Google Drive, giving them just the right access, to just the right files.
  • When people’s roles change on a project team, their Google Docs access is automatically and instantly changed.
  • When a document is renamed in Kerika, the new name is reflected back in Google Drive.

Many of our users who are already using Google Docs loved our integration: one user went so far as to say “Kerika makes Google Docs sing!”

The Google integration was not easy, particularly in the early days when there were wide gaps between the documentation and the reality of the API: we had to frequently resort to the wonderful Seattle Tech Startups mailing list to cast about for help.

But it seemed worth while: our Google integration has definitely helped us get paying customers — organizations moving off the traditional Microsoft Office/Exchange/SharePoint/Project stack were looking for a tool just like Kerika, particularly if they were also grappling with the challenges of becoming a Lean/Agile organization and managing an increasingly distributed workforce.

We even signed up organizations, like the Washington State Auditor’s Office who started using Google Apps for Government just because they wanted to use Kerika!

But, there are other folks we encounter all the time who say to us: “We love everything about Kerika except for the Google integration”

Some folks want to work with Microsoft Office file format all the time (although that’s possible even with our Google Drive integration, by setting a personal preference, and will be even easier in the future with new edit functions announced at Google I/O), but, more commonly, we came up against a more basic concern — people simply distrusted Google on privacy grounds.

It’s debatable as to whether it’s a well-grounded fear or not, but it is certainly a widespread fear, and it is not showing any signs of diminishing as we continue to talk to our users and prospects.

Some of this is due to a lack of understanding: users frequently confuse “security” and “privacy”, and tell us that they don’t want to use Google Apps because it isn’t secure. This is really far off the mark, for anyone who knows how Google operates, and understands the difference between security and privacy.

Google is very secure: more secure than any enterprise is likely to be on its own. They have a lot of software and a lot of people constantly looking for and successfully thwarting attackers. It’s always possible for someone to hack into your Google account, but it will be through carelessness or incompetence on your end, rather than a failure on Google’s part.

Privacy, however, is a different matter altogether, and here Google does itself no favors:

  • It’s Terms of Use are confusing: their general terms of use, for all Google services, contains this gem which drives lawyers crazy: “When you upload, submit, store, send or receive content to or through our Services, you give Google (and those we work with) a worldwide license to use, host, store, reproduce, modify, create derivative works (such as those resulting from translations, adaptations or other changes we make so that your content works better with our Services), communicate, publish, publicly perform, publicly display and distribute such content.”
    The Google Apps for Business Terms are much more specific: “this Agreement does not grant either party any rights, implied or otherwise, to the other’s content or any of the other’s intellectual property.”
    But most people derive a first, and lasting, impression from Google general terms, and few get around to investigating the GAB specific terms.
  • Google finds it hard to acknowledge people’s privacy concerns. This is somewhat puzzling, and can perhaps be best explained as a cultural problem. Google genuinely thinks itself of a company that “does no evil”, and therefore finds itself reflexively offended when people question its commitment to privacy. It’s hard to address a problem that challenges your self-identity: the Ego acts to protect the Id.

Entire sectors seem closed to Google Drive: lawyers, who could certainly benefit from Kerika’s workflow management, and healthcare, which is already adopting Lean techniques (pioneered locally by the Virginia Mason hospitals.)

In the small-medium business (SMB) market in particular, there isn’t any meaningful outreach by Google to address the privacy/security concern.  (Google does reach out to large enterprises, but in the SMB market it relies entirely on resellers.)

For our part, we have done a ton of work persuading people that it’s OK to use Google Drive, but we don’t get paid for this (we are not a Google Apps reseller), and this is, at best, a distraction from our core mission of building the very best work management software for distributed, lean and agile teams.

We need an alternative cloud storage platform: one that has robust capabilities, is enterprise-friendly, and doesn’t come with any privacy baggage.

The full series:

 

Why we are integrating with Box; Part 1: Privacy Overhang

Kerika, Technology,

(The first in a series of blog posts explaining why, and how, we are adding Box as a cloud service in addition to our long-standing integration with Google Drive.)

When we first started working on Kerika, back in 2010, Google Docs was an obvious choice for us to integrate with: it was pretty much the only browser-based office suite available (Microsoft Office 365 wasn’t around and Zoho was, and remains, somewhat obscure), and we were quite sure we didn’t want to get in the business of storing people’s desktop files in the cloud.

Google Docs (not yet renamed Google Drive) did have various limitations in terms of the file types it would support, and further limitations on the maximum size permitted of different types of files — the largest spreadsheet that they would support, for example, wasn’t the same size as the largest Word document — but the idea of building Kerika as a pure-browser app was a very appealing one.

So we integrated with Google Docs at a low level, fully embracing their API, to provide a seamless experience unlike anything that you might find from competing products:

  • When you add a document to a card or canvas in a Kerika board, it is automatically uploaded to your Google Drive, where it is stored in a set of nested folders.
  • When you add people to a Kerika board, their Google Drives automatically get dynamic links to your Google Drive, giving them just the right access, to just the right files.
  • When people’s roles change on a project team, their Google Docs access is automatically and instantly changed.
  • When a document is renamed in Kerika, the new name is reflected back in Google Drive.

Many of our users who are already using Google Docs loved our integration: one user went so far as to say “Kerika makes Google Docs sing!”

The Google integration was not easy, particularly in the early days when there were wide gaps between the documentation and the reality of the API: we had to frequently resort to the wonderful Seattle Tech Startups mailing list to cast about for help.

But it seemed worth while: our Google integration has definitely helped us get paying customers — organizations moving off the traditional Microsoft Office/Exchange/SharePoint/Project stack were looking for a tool just like Kerika, particularly if they were also grappling with the challenges of becoming a Lean/Agile organization and managing an increasingly distributed workforce.

We even signed up organizations, like the Washington State Auditor’s Office who started using Google Apps for Government just because they wanted to use Kerika!

But, there are other folks we encounter all the time who say to us: “We love everything about Kerika except for the Google integration”

Some folks want to work with Microsoft Office file format all the time (although that’s possible even with our Google Drive integration, by setting a personal preference, and will be even easier in the future with new edit functions announced at Google I/O), but, more commonly, we came up against a more basic concern — people simply distrusted Google on privacy grounds.

It’s debatable as to whether it’s a well-grounded fear or not, but it is certainly a widespread fear, and it is not showing any signs of diminishing as we continue to talk to our users and prospects.

Some of this is due to a lack of understanding: users frequently confuse “security” and “privacy”, and tell us that they don’t want to use Google Apps because it isn’t secure. This is really far off the mark, for anyone who knows how Google operates, and understands the difference between security and privacy.

Google is very secure: more secure than any enterprise is likely to be on its own. They have a lot of software and a lot of people constantly looking for and successfully thwarting attackers. It’s always possible for someone to hack into your Google account, but it will be through carelessness or incompetence on your end, rather than a failure on Google’s part.

Privacy, however, is a different matter altogether, and here Google does itself no favors:

  • It’s Terms of Use are confusing: their general terms of use, for all Google services, contains this gem which drives lawyers crazy: “When you upload, submit, store, send or receive content to or through our Services, you give Google (and those we work with) a worldwide license to use, host, store, reproduce, modify, create derivative works (such as those resulting from translations, adaptations or other changes we make so that your content works better with our Services), communicate, publish, publicly perform, publicly display and distribute such content.”
    The Google Apps for Business Terms are much more specific: “this Agreement does not grant either party any rights, implied or otherwise, to the other’s content or any of the other’s intellectual property.”
    But most people derive a first, and lasting, impression from Google general terms, and few get around to investigating the GAB specific terms.
  • Google finds it hard to acknowledge people’s privacy concerns. This is somewhat puzzling, and can perhaps be best explained as a cultural problem. Google genuinely thinks itself of a company that “does no evil”, and therefore finds itself reflexively offended when people question its commitment to privacy. It’s hard to address a problem that challenges your self-identity: the Ego acts to protect the Id.

Entire sectors seem closed to Google Drive: lawyers, who could certainly benefit from Kerika’s workflow management, and healthcare, which is already adopting Lean techniques (pioneered locally by the Virginia Mason hospitals.)

In the small-medium business (SMB) market in particular, there isn’t any meaningful outreach by Google to address the privacy/security concern.  (Google does reach out to large enterprises, but in the SMB market it relies entirely on resellers.)

For our part, we have done a ton of work persuading people that it’s OK to use Google Drive, but we don’t get paid for this (we are not a Google Apps reseller), and this is, at best, a distraction from our core mission of building the very best work management software for distributed, lean and agile teams.

We need an alternative cloud storage platform: one that has robust capabilities, is enterprise-friendly, and doesn’t come with any privacy baggage.

The full series:

 

A new template for tracking bugs and defects

Agile & Scrum, Kerika, Technology, ,

We just added another project template, for Kanban / Task Boards, that you can use to track and manage bugs/defects in a software or hardware product.

Bug Tracking Template
Click here to view this template

Here’s how it works:

  1. As new bugs/defects are found, add them to the Pending column. This is a holding area for new defects, before they get evaluated and prioritized.
  2. On a daily basis the Product Owner should evaluate items in the Pending column, considering both severity and priority.

    Severity is not the same as Priority: if a defect has a serious consequence, e.g. a software bug that causes data loss, then it would have a high Severity rating.

    But, some defects show up only rarely. So, you might have a defect with a serious consequence that happens very rarely, or affects very few users. In that case, you may want to reduce its priority.

    On the other hand, you may have a defect that is trivial, e.g. a confusing term on a website, but is a real annoyance for everyone. This bug could low severity, but high priority — because fixing it would immediately benefit a lot of your users.

  3. As the Product Owner evaluates each bug, it gets moved to the appropriate column: Fix Immediately, Fix Soon, or Deferred. Within each column, you can further prioritize bugs by moving the most important ones to the top of the column.

    Fix Immediately is for the most critical bugs; typically these need to be resolved with a day or two, and the software update may need to be delivered as a hotfix if the normal release cycle is too long.

    Fix Soon is for bugs that you definitely need to fix, but which are not super-critical.

    Deferred is for bugs that you are not likely to fix anytime soon. (If you don’t plan to fix a bug at all, move it to the Trash.)

  4. As each bug is picked up by a team member, move the card to In Progress, and assign the card to the team member: once someone’s face shows up on the card, it’s clear to everyone that the bug is being worked on, and by whom.
  5. As appropriate, use review processes to review and test the fix before moving it to Ready for Deployment.
  6. Use tags that make sense: we have set up some sample tags for this template; if they don’t make sense, use whatever tags will work best for your team! (There’s a short video on how tags work, attached to this card.)

Another round of bug bashing…

Bug Fixes, Kerika, Technology

Our current release (Sprint No. 37 since we moved to a Scrum methodology!) is focused on bug bashing: mostly server-related bugs, and a few that users might have observed themselves.

This is fairly typical of our development cycles: while we fix major bugs in every release, every once in a while we spend an entire release on just general bug bashing and cleanup.  These offer opportunities to catch up on infrastructure improvements, getting our test cases in better shape and other administrative work.

Release 37 should be done, with testing, by the end of the week: it will have about 65 work items completed, with another 21 items that were trashed.

Trashed items are usually duplicates of bugs: different symptoms with the same underlying cause.

Heartbleed: no heartache, but it did prompt a complete security review

Kerika,

So, here’s how we dealt with the Heartbleed bug…

We learned about the bug just like you did: through news reports early on April 7th. Heartbleed was a “zero-day” bug, and the OpenSSL team put out an updated (patched) version of the OpenSSL protocol the same day, which meant that everyone, everywhere, had to scramble to get their systems patched as quickly as possible.

(And the bad guys, meanwhile, scrambled to grab sensitive information, with the Canadian tax authorities being among the first to report that they had been hacked. Of course, “first to report” isn’t the same as “first to actually get hacked”. Most people who got hacked either never found out, or never said anything…)

Kerika uses OpenSSL too, and our immediate concern was updating the Elastic Load Balancer that we use to manage access to our main Amazon Web Services (AWS) servers: the load balancers are where OpenSSL is installed; not on the Web servers that sit behind the load balancer.

Using Amazon Web Services turned out to be a really smart decision in this respect: Amazon just went ahead and patched all their load balancers one by one, without waiting for their customers to take any action. In fact, they patched our load balancer faster than we expected!

Patching the load balancer provided critical immediate protection, and gave us the time to do a more leisurely security review of all our operations. This was long overdue, it turned out, and so we went into “housecleaning mode” for over a week.

One part of this, of course, was updating all our Ubuntu Linux machines: Canonical Software was also very prompt in releasing a patched version of Ubuntu which we loaded onto all of our development, test, and production services. So, even though the OpenSSL vulnerability had been patched at the load balancer, we also applied patches on all our development, test and production servers even though these couldn’t be directly accessed from the Internet.

Next, we decided to clean up various online services that we weren’t actively using: like many other startups, we frequently try out various libraries and third-party services that look promising. We stick with some; others get abandoned. We had accumulated some API keys for services that we weren’t using any more (e.g. we had a YouTube API key that no one could even remember why we had gotten in the first place!), and we deactivated everything that wasn’t actively been used.

Closing unneeded online accounts helped reduce our “attack surface”, which adds to our overall security.

And, of course, we changed all our passwords, everywhere. All of our email passwords, all of our third-party passwords. All of our online passwords and all of our local desktop passwords. (On a personal level, our staff also took the opportunity to change all their banking and other online passwords, and to close unneeded online accounts, to reduce our personal attack surfaces as well.)

We got new SSL certificates: from Verisign for our production load balancer, and from GoDaddy for our test load balancer. Getting a new SSL certificate from Verisign took much longer than we would have liked; getting one from GoDaddy took just seconds, but on the other hand, Verisign does have a better reputation…

We reviewed our internal security policies and procedures, and found a few places where we could tighten things up. This mostly involved increased use of two-party authentication and — most importantly — further tightening up access to various services and servers within the Kerika team. Access to our production servers is highly restricted even within the Kerika team: we use AWS’s Identity & Access Management service to restrict access using roles and permissions, even within the small subset of people who have any access to the production server.

Finally, we are adding more monitoring, looking out for malicious activity by any user, such as the use of automated scripts. We have seen a couple of isolated examples in the past: not malicious users, but compromised users who had malware on their machines. Fortunately these attempts were foiled thanks to our robust access control mechanisms which manage permissions at the individual object level in Kerika — but, like every other SaaS company, we need to be vigilant on this front.

All of this was good housekeeping. It disrupted our normal product development by over a week as we took an “all hands on deck” approach, but well worth it.

Heartbleed: no heartache, but it did prompt a complete security review

Kerika,

So, here’s how we dealt with the Heartbleed bug…

We learned about the bug just like you did: through news reports early on April 7th. Heartbleed was a “zero-day” bug, and the OpenSSL team put out an updated (patched) version of the OpenSSL protocol the same day, which meant that everyone, everywhere, had to scramble to get their systems patched as quickly as possible.

(And the bad guys, meanwhile, scrambled to grab sensitive information, with the Canadian tax authorities being among the first to report that they had been hacked. Of course, “first to report” isn’t the same as “first to actually get hacked”. Most people who got hacked either never found out, or never said anything…)

Kerika uses OpenSSL too, and our immediate concern was updating the Elastic Load Balancer that we use to manage access to our main Amazon Web Services (AWS) servers: the load balancers are where OpenSSL is installed; not on the Web servers that sit behind the load balancer.

Using Amazon Web Services turned out to be a really smart decision in this respect: Amazon just went ahead and patched all their load balancers one by one, without waiting for their customers to take any action. In fact, they patched our load balancer faster than we expected!

Patching the load balancer provided critical immediate protection, and gave us the time to do a more leisurely security review of all our operations. This was long overdue, it turned out, and so we went into “housecleaning mode” for over a week.

One part of this, of course, was updating all our Ubuntu Linux machines: Canonical Software was also very prompt in releasing a patched version of Ubuntu which we loaded onto all of our development, test, and production services. So, even though the OpenSSL vulnerability had been patched at the load balancer, we also applied patches on all our development, test and production servers even though these couldn’t be directly accessed from the Internet.

Next, we decided to clean up various online services that we weren’t actively using: like many other startups, we frequently try out various libraries and third-party services that look promising. We stick with some; others get abandoned. We had accumulated some API keys for services that we weren’t using any more (e.g. we had a YouTube API key that no one could even remember why we had gotten in the first place!), and we deactivated everything that wasn’t actively been used.

Closing unneeded online accounts helped reduce our “attack surface”, which adds to our overall security.

And, of course, we changed all our passwords, everywhere. All of our email passwords, all of our third-party passwords. All of our online passwords and all of our local desktop passwords. (On a personal level, our staff also took the opportunity to change all their banking and other online passwords, and to close unneeded online accounts, to reduce our personal attack surfaces as well.)

We got new SSL certificates: from Verisign for our production load balancer, and from GoDaddy for our test load balancer. Getting a new SSL certificate from Verisign took much longer than we would have liked; getting one from GoDaddy took just seconds, but on the other hand, Verisign does have a better reputation…

We reviewed our internal security policies and procedures, and found a few places where we could tighten things up. This mostly involved increased use of two-party authentication and — most importantly — further tightening up access to various services and servers within the Kerika team. Access to our production servers is highly restricted even within the Kerika team: we use AWS’s Identity & Access Management service to restrict access using roles and permissions, even within the small subset of people who have any access to the production server.

Finally, we are adding more monitoring, looking out for malicious activity by any user, such as the use of automated scripts. We have seen a couple of isolated examples in the past: not malicious users, but compromised users who had malware on their machines. Fortunately these attempts were foiled thanks to our robust access control mechanisms which manage permissions at the individual object level in Kerika — but, like every other SaaS company, we need to be vigilant on this front.

All of this was good housekeeping. It disrupted our normal product development by over a week as we took an “all hands on deck” approach, but well worth it.