upday tech blog - now at mediumJekyll2018-01-21T13:15:40+00:00https://upday.github.io/upday devhttps://upday.github.io/dev@upday.comhttps://upday.github.io/blog/we_are_moved2017-09-20T08:00:00+00:002017-09-20T08:00:00+00:00Peter Kraußhttps://upday.github.iopeter.krauss@upday.com
<h3 id="weve-moved-to-medium">We’ve moved to medium!!!!</h3>
<p>Basically the headline says it - we decided to move our blog over to <a href="https://medium.com/upday-devs">medium.com</a>.
Follow us there!!!</p>
<p><a href="https://upday.github.io/blog/we_are_moved/">We've moved</a> was originally published by upday dev at <a href="https://upday.github.io">upday tech blog - now at medium</a> on September 20, 2017.</p>
https://upday.github.io/blog/about_building_trust2017-09-18T22:00:55+00:002017-09-18T22:00:55+00:00Srikanth Achantahttps://upday.github.iosrikanth@upday.com
<p>Trust is the most precious resource in any organisation. For modern software organisations, which depend heavily on its employees for its success, trust becomes of paramount importance.</p>
<p>When trust is a topic, it’s clear that there is a deficit of it. It is a dangerous state to be in and it mainly affects the company in two ways. Increased cost of operations (by many fold) and shrinking space for creativity/innovation. One hurts the business in the short-to-medium term and the other hurts in the long term.</p>
<p>When the atmosphere in an organisation is one of distrust, people lose patience and empathy for others, and everything they dislike will be seen as a problem/failure.</p>
<p>If a feature is delayed it’s due to an incompetent engineer; if sales targets are not met it’s due to a poor tech setup; if the office looks empty it’s because engineers don’t take their work seriously;
if the vision or strategy is not clear it’s due to poor management; if the product roadmap lacks ‘interesting’ features it’s due to incompetent POs and so on. <strong>This is Stage One</strong>.</p>
<p>Although exaggerated, if you look closely there is some truth (among all the excuses) in these statements, but people refuse to accept and change. They are instead busy shifting the blame onto others.</p>
<p><strong>In Stage Two</strong>, engineers start adding more buffer to their estimates so they are never late, sales targets will be set so low that it’s hard to miss them, the office will look full but nothing gets done, fancy features are added to please people rather than users, workshops will be held to come up with catchy vision statements, etc, etc.</p>
<p>People stop taking risks and they become complacent. They add additional layers of insurance to their work to feel secure. All these activities take the time, money and energy of the team, which should otherwise be focused in achieving business goals.</p>
<p><strong>In Stage Three</strong>, the most dangerous phase of all, distrust becomes part of the culture.</p>
<p>Every new initiative, new decision, new idea, new process will be met with pessimism. As a result of it people disengage. It’s the worst thing that can happen to a company.</p>
<p><strong><em>How do you get out of this?</em></strong></p>
<p>The first step is to acknowledge it. There is no easy way out, everyone in the organisation needs to put in extra effort to overcome this.</p>
<p>Unless we change it, it’ll just be what it is.</p>
<p>Trust is a by-product of what you say and what you do. <em>So: work on improving in both these areas.</em></p>
<p><strong>Over-communicate:</strong> best way to address trust issues is to talk more, find ways to inform people about what is being worked on and what is not. If something troubles you, go talk to the concerned people and find out more. If there is something you disagree with, express your views constructively, rationally and respectfully.</p>
<p><strong>Expectations:</strong> be rational in what you expect and look back at achievements realistically. Take those rose-tinted glasses away.</p>
<p><strong>Show empathy:</strong> if you don’t understand something, try to learn more about it, don’t just jump to convenient conclusions. Incorporate people from other departments to share and gain knowledge.</p>
<p><strong>Good communication:</strong> with right expectation and empathy sets a foundation for people to build trust with each other. It is only the beginning of what should be a continuous process of improvement.</p>
<p>Trust is neither given nor taken, it’s built and maintained. One has to be aware of this at all time. It can never be taken for granted.</p>
<p>Teams can win if they work together, but they are sure to lose if they don’t.</p>
<p><a href="https://upday.github.io/blog/about_building_trust/">About Building Trust</a> was originally published by upday dev at <a href="https://upday.github.io">upday tech blog - now at medium</a> on September 19, 2017.</p>
https://upday.github.io/unpublished/po2017-09-07T01:00:00+00:002017-09-07T01:00:00+00:00upday devhttps://upday.github.iodev@upday.com
<h2 id="about-the-role">About the Role</h2>
<p>Data is core to upday’s advertising products, content recommendation as well as our
overall decision-making process. We are looking for a “Product Owner - Data &
Audiences” to lead a cross-functional team of data engineers and analysts. The Data
team owns data quality and infrastructure, builds data products, tackles audience &
product analysis and enables the data-driven culture across the organization.</p>
<p><strong>Your Profile</strong></p>
<ul>
<li>You have 2+ years experience as a Product Owner / in a product
management role working closely with Engineers/ Data Scientists</li>
<li>You have experience in managing/building data products/ BI solutions</li>
<li>You bring passion for data and building products</li>
<li>Proven track record of managing technical projects and delivering scalable,
production-grade products</li>
<li>Business Intelligence background is a plus</li>
<li>You are a excellent communicator and a true team player</li>
<li>You are experienced with and value agile methodologies</li>
<li>Very good spoken and written English</li>
</ul>
<p><strong>Your Job</strong></p>
<ul>
<li>Drive upday’s data & audiences vision and take ownership of the data
roadmap and it’s execution with your team.</li>
<li>Oversee and take responsibility over all our data products, analytics tools and
data infrastructure and make sure we work on the right setup.</li>
<li>Ensure accurate data is the number one priority in all that you and your team
deliver, and be prepared to ask the right questions to guarantee that it is.</li>
<li>Alongside the team, take it upon yourself to find your own insights and share
them in a way that creates a data-driven mentality throughout upday.</li>
<li>Identify ways to leverage our data across multiple business areas.</li>
<li>Own the team’s backlog and ensure the team’s focus on business value
creation.</li>
<li>Collaborate with the Product Team on the overall Product roadmap.</li>
<li>Clearly communicate the vision, plans and timelines to key stakeholders.</li>
<li>Engage in your team and help them grow. Work with the team to ensure we
have the right people and set up to achieve our goals.</li>
<li>Work from our office in Berlin.</li>
</ul>
<p><strong>What We Offer</strong></p>
<p>A position that offers a great deal of responsibility and impact</p>
<ul>
<li>The chance to collaborate with talented colleagues and develop new business
models for a content app</li>
<li>Being part of an international, very exciting and revolutionary project</li>
<li>A young, dynamic and fun team with experienced colleagues from different
departments of Axel Springer (tech, product, marketing, journalism)</li>
<li>The latest equipment and digital tools</li>
<li>Competitive salary and contract</li>
</ul>
<p>If you are interested in joining upday please <a href="https://global3.recruitmentplatform.com/syndicated/private/syd_apply.cfm?ID=PPKFK026203F3VBQB8N68V7HK&nPostingTargetID=8462">submit</a>
your application including cover letter, CV, your portfolio, earliest starting date, salary expectations and relevant
references</p>
<p><a href="https://upday.github.io/unpublished/po/">Product Owner Data & Audiences</a> was originally published by upday dev at <a href="https://upday.github.io">upday tech blog - now at medium</a> on September 07, 2017.</p>
https://upday.github.io/jobs/qa2017-09-06T04:00:55+00:002017-09-06T04:00:55+00:00upday devhttps://upday.github.iodev@upday.com
<p><strong>Your Profile</strong></p>
<ul>
<li>Process-minded problem solvers with a track record of creating environments
that help people work effectively</li>
<li>Skilled at identifying quality weaknesses in mobile apps, suggesting solutions
and driving change</li>
<li>Constantly looking for ways to improve quality and testing approach</li>
<li>Care deeply about quality and has great attention to detail</li>
<li>An excellent communicator used to juggle multiple teams with varying
agendas, and to coordinate and execute under pressure</li>
<li>Prioritize regularly and work iteratively to maximize positive organizational
impact</li>
<li>You have demonstrable experience of working on critical, highly visible
projects and thriving even when given a short timeline</li>
</ul>
<p><strong>Your Job</strong></p>
<ul>
<li>Set up QA processes to pay close attention to flag issues in an efficient
manner</li>
<li>Be an overall quality leader and constantly seek opportunities to improve
quality and testing approach across the organisations</li>
<li>Create and maintain internal service level agreements with stakeholders to
ensure new initiatives are flagged with the QA team on time</li>
<li>Define stakeholder relationships and cultivate strong networks across Content,
Tech, Product and Design to make sure everyone is on board</li>
<li>Work closely with developers within various tech team to cover all angles</li>
<li>Ensure bug and incident management processes are functional and
continuously maintained</li>
<li>Report bugs and doggedly pursue through the full cycle to effectively reach
the best possible solution.</li>
<li>Checkout and build the app and have an overview of the automated test suite</li>
<li>Collect learning’s and analyse problems in production to improve our proactive
process, thus decreasing the amount of unexpected issues</li>
<li>Increase transparency by capturing all known sub-optimal issues in a clear &
simple format. When possible, follow up with each team and track
improvements to ensure the problem has been amended in time for next
applicable content release</li>
</ul>
<p><strong>What We Offer</strong></p>
<ul>
<li>A position that offers a great deal of responsibility and impact</li>
<li>The chance to collaborate with talented colleagues and develop new business
models for a content app</li>
<li>Being part of an international, very exciting and revolutionary project</li>
<li>A young, dynamic and fun team with experienced colleagues from different
departments of Axel Springer (tech, product, marketing, journalism)</li>
<li>The latest equipment and digital tools</li>
<li>Competitive salary and contract</li>
</ul>
<p>If you are interested in joining upday please <a href="https://global3.recruitmentplatform.com/syndicated/private/syd_apply.cfm?ID=PPKFK026203F3VBQB8N68V7HK&nPostingTargetID=8408">submit</a>
your application including cover letter, CV, your portfolio, earliest starting date, salary expectations and relevant
references.</p>
<p><a href="https://upday.github.io/jobs/qa/">Senior QA Engineer</a> was originally published by upday dev at <a href="https://upday.github.io">upday tech blog - now at medium</a> on September 06, 2017.</p>
https://upday.github.io/jobs/devops2017-09-06T04:00:55+00:002017-09-06T04:00:55+00:00upday devhttps://upday.github.iodev@upday.com
<h2 id="about-the-role">About the Role</h2>
<p>You embrace the DevOps culture. You have clean code for breakfast, you domain drive to work, then you continuously integrate and release by lunchtime.
In the afternoon you get up-to-date with the latest technologies and before leaving, you improve the monitoring, add a circuit-breaker or do whatever is necessary to have a monkey-proof system you are proud of.
You go to bed with the DoD under your pillow and dream about orchestrating infrastructure with a smile on your face.</p>
<p><strong>Responsibilities:</strong></p>
<ul>
<li>Identify new technologies which can drive improvements in efficiency for production materials and systems</li>
<li>Consult/Provide/Improve stable easy to use “PaaS” / Infrastructure / app solution</li>
<li>Error analysis, Network troubleshooting</li>
</ul>
<p><strong>Qualifications and Skill-Set:</strong></p>
<ul>
<li>Experience in at least one of the following languages: Python, Ruby, Bash</li>
<li>Proficiency in one automation tool: Ansible (preferred), Chef, Puppet</li>
<li>Worked with cloud services provider: AWS (preferred), GCE, Azure</li>
<li>Continuous Integration and Deployment: Jenkins</li>
<li>Familiar with centralized logging and monitoring: Splunk, Datadog</li>
<li>Good Linux System Administration knowledge</li>
<li>Plus: RabbitMQ, Elasticsearch, Postgres</li>
</ul>
<p>Experience in any of the below would be a plus:</p>
<ul>
<li>Docker</li>
<li>Infrastructure testing</li>
<li>Spring Boot</li>
</ul>
<p><strong>Your Profile</strong></p>
<ul>
<li>You are a team player with experience working in an agile environment</li>
<li>You are keen to develop your skills, and those of your team</li>
<li>You have an interesting hobby such as brewing beer or space travel</li>
</ul>
<p>We expect that development is not new to you and will ask you to show us your skills by taking part in a test to demonstrate your expertise.</p>
<p>You may be young or you may be a veteran in the software development world. This is not important to us.</p>
<p>If this is you, <a href="mailto:jobs@updudes.net">get in touch with us and send us an email to jobs@updudes.net</a>!</p>
<p><a href="https://upday.github.io/jobs/devops/">DevOps Engineer</a> was originally published by upday dev at <a href="https://upday.github.io">upday tech blog - now at medium</a> on September 06, 2017.</p>
https://upday.github.io/blog/upday-be-architecture2017-08-22T22:55:55+00:002017-08-22T22:55:55+00:00María Fernández Pajareshttps://upday.github.iomaria@upday.com
<p>I have always wanted to write about our journey through the world of backend architectures. As a developer, you may have been facing problems related to this topic in your team or with your projects.
You might even be aware of the magical and trendy solutions out there, but sometimes these don’t perfectly fit your current needs.</p>
<p>This topic has not been - and it is not yet - an easy one for us either, but after facing many problems we now find ourselves in the world of backend architecture.
Therefore I would like to share a few of our experiences and learnings in this area at upday.</p>
<h3 id="conways-laws-reversion">Conway’s Law’s reversion</h3>
<p>In conferences I have attended I have heard references to what could be interpreted from <a href="https://en.wikipedia.org/wiki/Conway%27s_law"><em>Mel Conway’s Law</em></a>:</p>
<blockquote>
<p>A backend architecture is a reflection of the team setup.</p>
</blockquote>
<p>and I totally agree with that but I also think that <strong>with time it may become the other way around:</strong></p>
<blockquote>
<p>the architecture can end up shaping the team instead of the team shaping the architecture.</p>
</blockquote>
<p>Let me dig more into the challenges we face this time around and how we managed to resolve them:</p>
<h3 id="our-firsts-architectural-steps">Our firsts architectural steps</h3>
<p>At <em>upday</em> we started with a small team of developers tasked with creating a robust solution to fulfill our client’s requirements of reliability and providing good quality content to our readers.
As in most <em>startups</em>, everything started fast and in a rush. The solution we finally came up with consisted of two monolithic components written in <em>Java</em> and <em>Spring Boot</em>.
Together they form the core of our system. But they were not the only ones. We also had other smaller components written in Ruby or Java, which took care of minor requirements.
When this setup worked properly and was production ready, we went live with it.</p>
<h3 id="when-our-architecture-started-to-have-flows">When our architecture started to have flows</h3>
<p><strong>When our company started growing, it was impossible to stick to only one big team of backend developers.</strong></p>
<p>We started splitting our backend team into feature teams. This was not an optimal setup at that time, given these two big components had to be touched by each team every time a new feature was added.</p>
<p><strong>When you have something stable and working, it is scary to start rewriting it but it’s important to do so sooner rather than later to avoid getting to the point of no return.</strong></p>
<p>With this in mind, we started to step into the world of <em>microservices</em>, to make life easier for us developers (and of course more challenging, fun and efficient).
For sure there’s a difference in adding one small field to a 50,000-lines-of-code monster and touching 1,000 different unit and integration tests, than doing the same in a small service.
The reviewer will be the first one to be thankful for that.</p>
<h3 id="where-to-draw-the-line-on-diversity-of-technology">Where to draw the line on diversity of technology</h3>
<p>Once living in this wonderful and interesting microservices world, <strong>it was nice to have the opportunity to use new technologies</strong> without being restricted to just the original ones.
However, not everything is pink and full of rainbows in the microservices world. <strong>You have to know when to stop</strong> creating new microservices or adding more and more new technologies to your stack.
Because your team - whose members could always change - has to continue maintaining and understanding this zoo of services and its underlying technologies.</p>
<p>Like everything in life: <strong>You have to find a balance.</strong> Since everyone has different opinions, there is no one perfect solution.
Also, something that is <em>en vogue</em> now may not be the <em>silver bullet</em> forever.
It might be that the best approach for your team is having less services and only one single technology.
Or it may be that applying different technologies and trying new ones is what the developers want to do, because they are hungry to learn and face new challenges.
<strong>Whatever the situation is, just don’t lose the focus on what your product really requires.</strong></p>
<h3 id="our-architecture-today">Our architecture today</h3>
<p>As a team of young developers who strive for challenges, we have evolved into our current setup where we have better small services.
<strong>But we never create a new microservice for the sake of creating a microservice</strong>, only when there are reasons to go for a new one.
And this seems to work quite well for both, our developers and our product.</p>
<p>Our <em>Android</em> app we serve has two main parts:</p>
<ul>
<li>the <em>Top News section</em> (news everyone needs to know about) and the</li>
<li><em>My News section</em> (based on the user’s interests and their interactions with our app).</li>
</ul>
<p>We decided to keep these areas separated and independent, <strong>basing their communication only on messaging</strong>, as described in the <strong>SCS</strong> <a href="http://scs-architecture.org/"><em>(Self Contained Systems)</em></a> approach.
So when there is any problem related with <em>Top News</em>, <em>My News</em> is not affected and the other way around.</p>
<p>This is how our architecture looks like from a bird’s eyes view (very high level overview):</p>
<p><img style="margin: auto; margin-left: 15%; margin-top: 10px;" src="/images/blog/upday_architecture/high-level_arch_overview.jpg" /><br /></p>
<p>And here a more detailed overview of the (micro-) services - not always so <em>“super micro”</em> - developed with different technologies, composing the <strong>Top News System</strong>:</p>
<p><img style="margin: auto; margin-left: 2%; margin-top: 10px;" src="/images/blog/upday_architecture/microservices_top_news_system.jpg" /><br /></p>
<h3 id="our-learnings-and-conclusions-so-far">Our learnings and conclusions so far</h3>
<p><em>… which are not universal truth and may not even last forever for us:</em></p>
<ul>
<li>Although we have mastered Java 8 and Spring Boot technologies so far, <strong>we might move to other technologies if they fit our requirements or problems better.</strong> But this learning has a big BUT:
<ul>
<li>we will only go for a new technology if more than one developer masters it, or in other words, the team gets involved and learns the new technology used in the project.</li>
</ul>
</li>
<li>
<p>Handling different types of data differs per service:</p>
<ul>
<li>for score-based querying of full text search, <em>Elasticsearch</em> comes handy</li>
<li>for getting statistics of open rates for our content team, using <em>AWS Lambdas</em> and <em>DynamoDB</em> fits much better</li>
<li>for simply storing articles with simple relational queries we use <em>Postgres</em></li>
</ul>
</li>
<li>And now we are quite impressed by <em>Kotlin</em>. Our Android team is already starting to use it and perhaps we too start to implement future backend services with it. Of course after analysing and doing pro-cons
of what it would mean for us developers and our product.</li>
</ul>
<p>This was my/our story so far.</p>
<p><strong>Good luck on your own path through backend architecture!</strong></p>
<p><a href="https://upday.github.io/blog/upday-be-architecture/">Finding yourself in the world of backend architecture</a> was originally published by upday dev at <a href="https://upday.github.io">upday tech blog - now at medium</a> on August 23, 2017.</p>
Part 2: We Need Storage]]>https://upday.github.io/blog/dwh-part2-we-need-storage2017-08-22T04:39:55+00:002017-08-22T04:39:55+00:00Robert Bordohttps://upday.github.iorobert@upday.com
<blockquote>
<p>In the beginning we created a cluster. And the cluster was without form, and void; and nulls were upon the face of the storage.</p>
</blockquote>
<p>As we learned in <a href="../dwh-part1_getting_started">part 1 of our series</a>, a data warehouse consists of several components.
The key component is the storage. All the others group around it. But how can one draw a decision on which storage solution to adopt? What is out there anyway?</p>
<h2 id="preface">Preface</h2>
<p>In a perfect world there would be only one kind of storage that fits all the needs of current DWH development and analysis.
But since we are not living in that kind of place, we have several options. And the number of options increase the deeper one dives into the topic.
There seem to be solutions for every use case you can think of.</p>
<p>That might be a good starting point. What is our most common use case? What are we going to store? And how would we like to access our data in the end?</p>
<p>Our major source is a massive amount of log data coming from our <a href="https://play.google.com/store/apps/details?id=de.axelspringer.yana">app</a>.
Everything the user does (e.g swiping through articles, selecting categories, leaving the app) is tracked,
enriched with metadata (e.g. the user’s location, app version, article identifier) and stored by a third-party service in big, semi-structured log files.
Having only this source, a time series database like
<a href="github.com/graphite-project/graphite-web">Graphite</a>
or
<a href="www.influxdata.com/time-series-platform/influxdb">InfluxDB</a>
could do the job.
But also having slow changing master data, like user profiles, article metadata and maybe even to keep a history of data, this solution would not satisfy our current and future needs.</p>
<p>Another thing that comes to my mind is how the data will be accessed by our final consumer (namely: Business Intelligence).
Usually they use tools like
<a href="https://en.wikipedia.org/wiki/JasperReports">Jasper Reports</a>
or
<a href="https://en.wikipedia.org/wiki/Tableau_Software">Tableau</a>
for generating reports.
For analyses we have to pre-aggregate the data to make queries more performant and translate raw information into a digestible format.</p>
<p>What else is on the market?</p>
<table>
<thead>
<tr>
<th>Storage</th>
<th>good at</th>
<th>bad at</th>
<th>Example</th>
</tr>
</thead>
<tbody>
<tr>
<td><strong>S3/Flat Files</strong></td>
<td>scalability, easy to use, data lake</td>
<td>querying</td>
<td>S3</td>
</tr>
<tr>
<td><strong>Time Series DB</strong></td>
<td>handling time series data</td>
<td>non time series data</td>
<td>Graphite</td>
</tr>
<tr>
<td><strong>Relational DB</strong></td>
<td>referential integrity, transactions</td>
<td>semi-structured data</td>
<td>Postgres</td>
</tr>
<tr>
<td><strong>Key Value Store</strong></td>
<td>semi-structured data, speed</td>
<td>complex queries</td>
<td>DynamoDB</td>
</tr>
<tr>
<td><strong>Document Store</strong></td>
<td>schema free</td>
<td>aggregating</td>
<td>Elasticsearch</td>
</tr>
<tr>
<td><strong>Wide Column Store</strong></td>
<td>storing large number of dynamic columns</td>
<td> </td>
<td>Hadoop</td>
</tr>
<tr>
<td><strong>Graph Database</strong></td>
<td>processing of graph structures</td>
<td>indexing</td>
<td>Neo4J</td>
</tr>
<tr>
<td><strong>Object Database</strong></td>
<td>storing objects</td>
<td>speed</td>
<td>Versant</td>
</tr>
</tbody>
</table>
<p><br />
And there are even more. An extensive list can be found <a href="http://nosql-database.org/">here</a>.</p>
<h2 id="the-given-one---s3">The Given One - S3</h2>
<p>Something I forgot to mention is, we store all (semi-structured) data we receive from different sources to S3 first.
S3 serves as our <a href="https://en.wikipedia.org/wiki/Data_lake">Data Lake</a>. This enables us to only load relevant data selectively to our main storage without losing information,
because we could re-load afterwards. This reduces costs (S3 is cheap, our main storage is not) and
<a href="https://aws.amazon.com/de/athena/">Amazon Athena</a> even enables us to query data in S3 directly.</p>
<h2 id="the-candidate---aws-redshift">The Candidate - AWS Redshift</h2>
<p>Most of our infrastructure is hosted on Amazon Web Services (AWS). It would be handy to stay on this well known ground.
There must be something out there. And it is indeed. <a href="https://aws.amazon.com/redshift">Redshift</a> is Amazon’s “data warehouse as a service” solution.</p>
<p>On Redshift’s homepage you can read:</p>
<p><em>Amazon Redshift is a fast, fully managed data warehouse that makes it simple and cost-effective to analyze all your data using standard SQL and your existing Business Intelligence (BI) tools.</em></p>
<p>Basically it is a column oriented database with good support for analytical functions. It is able to handle Petabytes of structured data by massively parallel query execution. Sounds good.
Fulfills our requirements. Why not give it a chance?</p>
<p><strong>Pros:</strong></p>
<ul>
<li>(unlimited) scalability</li>
<li>good support for analytical functions</li>
<li>accessible like any other database</li>
<li>good integration in AWS infrastructure (S3, AWS Lambdas, etc.)</li>
<li>good support to import data from S3</li>
</ul>
<p><strong>Cons:</strong></p>
<ul>
<li>Costly (depending on the type and number of nodes used)</li>
<li>steep learning curve in order to boost performance</li>
<li>maybe training necessary (or even consultancy)</li>
<li>no native support for semi-structured data</li>
<li>database administration is complex (column compression, distribution, vacuumization, …)</li>
</ul>
<h2 id="the-opponent---snowflake">The Opponent - Snowflake</h2>
<p>Redshift is not the only data warehouse as a service candidate. <a href="https://www.snowflake.net/">Snowflake</a> is another one.
To be honest, we evaluated Snowflake a while ago.
A lot has changed since then, but let’s stick to the data we collected sometime in 2016.</p>
<p>On Snowflake’s homepage you can read:</p>
<p><em>From zero to Petabytes, our elastic data warehouse architecture scales to any volume of data painlessly and inexpensively.
No more painful decisions about what data you can afford to keep.</em></p>
<p><strong>Pros:</strong></p>
<ul>
<li>storage is cheap (maybe) -> see cons</li>
<li>native support for semi-structured data</li>
<li>(unlimited) scalability</li>
<li>good integration into AWS infrastructure (S3, AWS Lambdas, etc.)</li>
<li>database administration is done automatically</li>
</ul>
<p><strong>Cons:</strong></p>
<ul>
<li>nontransparent pricing policy</li>
<li>missing information about tooling (ETL)</li>
</ul>
<p>Snowflake created a database layer on top of S3 (a little bit like Amazon Athena did later). Unfortunately it wasn’t easy to thoroughly test Snowflake at that time.
Consequently we did not investigate further. Maybe we would handle this issue differently today.</p>
<h2 id="finally">Finally</h2>
<p><strong>We decided for Redshift in the end.</strong> I have to admit it hasn’t been a too unbiased process, because more or less we wanted to try Redshift anyway.</p>
<p><strong>Starting with Redshift is easy, but getting satisfying results, especially if the amount of data increases, takes longer.</strong>
Execution time and costs increase. You have to understand how Redshift works under the hood and you need to tune your tables accordingly.
Also its maintenance is not a piece of cake. We will share our findings in a later post.</p>
<h2 id="summary">Summary</h2>
<p><strong>Selecting a proper storage for a data warehouse is tough.</strong> And it’s not at the same time. At the end of the day <strong>you, as a data engineer, manage data, not a database</strong>.
Data goes into the storage and is being retrieved again. Make sure both ways are comfortable. You, as a data engineer, are able to load data into the storage easily.
Your customer, usually some BI person, can fetch data quickly and is happy. Data maintenance should not be too complex as well. And all aggregations in-between should also work smoothly.</p>
<p><strong>Questions you should ask:</strong></p>
<ul>
<li>How does my infrastructure look so far?</li>
<li>What are my sources?</li>
<li>How is my source data structured?</li>
<li>How much data do I expect to store?</li>
<li>Do I need unlimited scalability?</li>
<li>What tools (BI) need to access my data?</li>
<li>How much can I spend on it?</li>
<li>How complicated is the maintenance?</li>
<li>Do we have the capacity and knowledge to master this technology?</li>
<li>Do we need training?</li>
</ul>
<p>And there’s nothing wrong with having more than one storage solution. I mean, storage is not Highlander in the end.</p>
<p><a href="https://upday.github.io/blog/dwh-part2-we-need-storage/">A Journey Towards a Custom Data Warehouse Solution<br/> Part 2: We Need Storage</a> was originally published by upday dev at <a href="https://upday.github.io">upday tech blog - now at medium</a> on August 22, 2017.</p>
https://upday.github.io/blog/360_andev2017-08-17T05:00:00+00:002017-08-17T05:00:00+00:00Peter Kraußhttps://upday.github.iopeter.krauss@upday.com
<p>As part of my <strong>Silicon Valley Fellowship</strong> from <strong>Axel-Springer</strong> I had the chance to visit the <strong>360|Andev</strong> and I would like to present you some of
the talks I attended.</p>
<h2 id="day-1">Day 1</h2>
<h4 id="making-your-app-instant---by-kasra-rahjerdi-">“Making your App Instant” - by <a href="https://twitter.com/jc4p">Kasra Rahjerdi </a></h4>
<p><a href="https://academy.realm.io/posts/360-andev-2017-kasra-rahjerdi-making-your-app-instant/">video</a></p>
<p><em>Kasra Rahjerdi</em> is the lead Android Developer at Stack Overflow and told us the story about how they implemented Android Instant Apps
for StackExchange in collaboration with Google. He highly recommended the <a href="https://codelabs.developers.google.com/codelabs/android-instant-apps/#0">codelab</a>
that Google provides. It is a tricky business to split an, app that was implemented as a monolith, into modules that can be reused.
He emphasized that you should <strong>never try to refactor your code while splitting it</strong>. The complexity is simply too high
and you will find yourself in a dead end! Split it, make it compile and if you feel so, make it more beautiful - but never the other way around.
I enjoyed the talk from <em>Kasra</em> very much, because he is a real energetic speaker who understands to entertain his audience.</p>
<center>
<picture>
<img src="/images/blog/andev/instant.png" alt="Making your App Instant" />
</picture>
</center>
<p><br /></p>
<h4 id="android-developer-options-deep-dive---by-andrea-falcone">“Android Developer Options Deep Dive” - by <a href="https://twitter.com/asfalcone">Andrea Falcone</a></h4>
<p>Andrea gave a great talk about the developer options of Android. I am working with this platform for round about 5 years now,
but I did not know about all the powerful tools.</p>
<ul>
<li><a href="https://developer.android.com/studio/debug/bug-report.html"><strong>Interactive Bug Report</strong></a> - interactive bug reports with
embedded screenshots - makes the reproduction of bugs much easier.</li>
<li><strong>wait for debugger</strong> - Have you ever had a bug in your <code class="highlighter-rouge">onCreate()</code> method? Then this is your tool! If you turn this option on, the
debugger will be attached before the app starts and you are able to debug the beginning of the lifecycle.</li>
<li><strong>animation scale</strong> - I guess most of us have used this tool already to turn of animations, because they are problematic
for <em>espresso</em> tests. But the tool is much more powerful. It allows you to change the duration of animations.
If you set it to <em>Animation scale 10x</em> the animation will be speed up by the factor 10.</li>
</ul>
<p>This tweet from <a href="https://twitter.com/KellyShuster">Kelly Schuster</a> shows the complete
content of the talk with the help of an awesome <em>sketchnote</em>.</p>
<center>
<blockquote class="twitter-tweet" data-lang="de"><p lang="en" dir="ltr">Such an amazing talk by <a href="https://twitter.com/asfalcone">@asfalcone</a> on the Android developer options!! I almost ran out of room 😅 <a href="https://twitter.com/hashtag/360andev?src=hash">#360andev</a> <a href="https://t.co/26SpAoaze7">pic.twitter.com/26SpAoaze7</a></p>— Kelly Shuster (@KellyShuster) <a href="https://twitter.com/KellyShuster/status/885615344320040960">13. Juli 2017</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
</center>
<p><br /></p>
<h4 id="there-is-room-for-improvement---by-florina-muntenescu">“There is Room for improvement” - by <a href="https://twitter.com/FMuntenescu">Florina Muntenescu</a></h4>
<p><a href="https://academy.realm.io/posts/360-andev-2017-florina-muntenescu-data-persistence-android-room/">video</a></p>
<p>Of course this was my favourite talk! It has been awesome to see Florina, our former “updudette” and greatest advocate of upday,
as a Google Developer Advocate now. Google published the <a href="https://developer.android.com/topic/libraries/architecture/index.html">Android Architecture Components</a>
at the I/O 2017 which also includes a persistence API called <a href="https://developer.android.com/topic/libraries/architecture/room.html">Room</a>.
Room is not an ORM. Instead it is a wrapper that simplifies the usage of <em>SQLite</em> in Android. Therefore it provides convenient
APIs to define table structures, relationships and migrations. Next to this it gives you compile time checks of <em>SQLite Statements</em> and
can return <em>RxJava</em> based <em>Flowables</em> and <em>Observable</em>s or <a href="https://developer.android.com/topic/libraries/architecture/index.html">LiveData</a>
Observables. The reactive nature makes it nicely testable.</p>
<center>
<picture>
<img src="/images/blog/andev/room.jpg" alt="Making your App Instant" />
</picture>
</center>
<p>Awesome sketchnote from <a href="https://twitter.com/chiuki">Chiu-Ki Chan</a> about Florinas talk:</p>
<center>
<blockquote class="twitter-tweet" data-lang="de"><p lang="en" dir="ltr">Data Persistence in Andorid. There is Room for improvement by <a href="https://twitter.com/FMuntenescu">@FMuntenescu</a> <a href="https://twitter.com/hashtag/360AnDev?src=hash">#360AnDev</a> <a href="https://twitter.com/hashtag/sketchnotes?src=hash">#sketchnotes</a> <a href="https://t.co/tXwMLZ0Vem">pic.twitter.com/tXwMLZ0Vem</a></p>— Chiu-Ki Chan (@chiuki) <a href="https://twitter.com/chiuki/status/885634916603502592">13. Juli 2017</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
</center>
<p><br /></p>
<h4 id="social-event">Social Event</h4>
<p>The <em>andev</em> team invited us to the <a href="http://mellowmushroom.com/store/downtown-denver">Mellow Mushroom</a> in downtown Denver, for some awesome Pizza
and a lot of different styles of beer. It is a nice feeling if you realize that you chat with <em>Chet Haase</em> about how to give talks and how to answer questions
in the best way. It was just amazing to see how open this community is!</p>
<center>
<picture>
<img src="/images/blog/andev/pizza.jpg" alt="Social Event" />
</picture>
Florina and me at the social event
</center>
<hr />
<h2 id="day-2">Day 2</h2>
<h4 id="dont-fear-sql---by-leandro-favarin">“Don’t fear SQL” - by <a href="https://twitter.com/leandrofavarin">Leandro Favarin</a></h4>
<p><a href="https://speakerdeck.com/leandrofavarin/dont-fear-sql-360andev-2017">slides</a> and <a href="https://academy.realm.io/posts/360-andev-2017-leandro-favarin-sqlbrite-sqdelight/">video</a></p>
<p><em>Leandro</em> is the Lead Android Developer of the Berlin based startup <a href="http://www.blinkist.com">Blinkist</a>.
His talk was about the benefits of <a href="https://github.com/square/sqldelight">SQLDelight</a> and <a href="https://github.com/square/sqlbrite">SQLBrite</a>
which are both libraries provided by <em>Square</em>. <em>SQLDelight</em> generates Java models from SQL statements that you put into a <code class="highlighter-rouge">.sq</code> file.
By utilising the power of SQL it gives you all the freedom to define every query that you could come up with. With the words
of Jake Wharton: <strong>“SQL should be embraced, not hidden”</strong>.</p>
<p>It becomes tricky if you need to do migrations of your tables with <em>SQLDelight</em>, because it isn’t officially supported yet. If you wanna follow the progress,
there is an open issue on <a href="https://github.com/square/sqldelight/issues/89">Github</a>. The <a href="https://plugins.jetbrains.com/plugin/8191-sqldelight">IntelliJ Plugin</a> for <em>SQLDelight</em>
includes syntax highlighting, code autocompletion and shows compiler errors at build time.</p>
<p><a href="https://github.com/square/sqlbrite">SQLBrite</a> on the other hand allows you to introduce reactive stream functions to your queries.
Definitely check out <em>Leandro’s</em> slides if you need to deal with a database in your app.</p>
<center>
<blockquote class="twitter-tweet" data-lang="de"><p lang="en" dir="ltr">SQL should be embraced, not hidden. Real nice talk from <a href="https://twitter.com/leandrofavarin">@leandrofavarin</a> from <a href="https://twitter.com/blinkist">@blinkist</a> at <a href="https://twitter.com/hashtag/360andev?src=hash">#360andev</a> <a href="https://t.co/TR8gB7Awtz">pic.twitter.com/TR8gB7Awtz</a></p>— Peter (@pkatberlin) <a href="https://twitter.com/pkatberlin/status/885890358059515908">14. Juli 2017</a></blockquote>
<script async="" src="//platform.twitter.com/widgets.js" charset="utf-8"></script>
</center>
<p><br /></p>
<h4 id="rx-by-example---the-multicast-edition-vol-3---by-kaushik-gopal">“RX by example - The Multicast edition Vol 3” - by <a href="https://twitter.com/kaushikgopal">Kaushik Gopal</a></h4>
<p><a href="https://speakerdeck.com/kaushikgopal/rx-by-example-volume-3-the-multicast-edition">slides</a> and <a href="https://academy.realm.io/posts/360-andev-2017-kaushik-gopal-rxjava-by-example-multicasting/">video</a></p>
<p>If you ever had the feeling that <em>RxJava</em> is awesome but pretty tough to learn - well you’re not alone. But no worries <em>Kaushik</em> comes to the rescue.
He presented the third part of his <em>“RX by example”</em> series in Denver. Volume 3 shows you how to <em>Multicast</em> events to several subscribers and when to use
<code class="highlighter-rouge">publish()</code>, <code class="highlighter-rouge">refCount()</code> and <code class="highlighter-rouge">autoConnect()</code> or in which permutations. He did an awesome job by visualizing this
complex topic in an understandable manner - so go and checkout his <a href="https://speakerdeck.com/kaushikgopal/rx-by-example-volume-3-the-multicast-edition">slides</a>!</p>
<center>
<picture>
<img src="/images/blog/andev/rx.jpg" alt="RX by example" />
</picture>
</center>
<p><br /></p>
<h4 id="finding-and-fixing-performance-problems---chet-haase-and-romain-guy">“Finding and Fixing Performance Problems” - <a href="https://twitter.com/chethaase">Chet Haase</a> and <a href="https://twitter.com/romainguy">Romain Guy</a></h4>
<p><a href="https://academy.realm.io/posts/360-andev-2017-romain-guy-chet-haase-android-performance/">video</a></p>
<p><em>Chet and Romain</em> showed by the example of the Google Play Store how to debug performance issues in an app. In this case there is
a notifiable lag while scrolling through the list of apps. They used the new profiler tool from <a href="https://developer.android.com/studio/preview/index.html">Android Studio 3</a>,
that allows a deep dive into all the layers and threads of the view elements. It can be controlled like a shooter game, by using w,a,s,d 😃.
By using the <a href="https://developer.android.com/studio/profile/hierarchy-viewer.html">Hierarchy viewer</a>
they figured out that the Play Store has way too many views to be rendered smoothly. Inside of the Android team they use <a href="https://developer.android.com/training/testing/performance.html">gfxinfo</a>
to measure the performance of the app after every build and visualize it in a dashboard over time. At <em>upday</em> we do a similar thing with our UI
tests, to discover flaky tests in the suite. Maybe we can use the same tech stack (Elasticsearch and Grafana) to visualize the performance of our app.</p>
<center>
<picture>
<img src="/images/blog/andev/perf.jpg" alt="Finding and Fixing Performance Problems" />
</picture>
</center>
<p><br /></p>
<h4 id="deep-android-integrations---by-ty-smith">“Deep Android Integrations” - by <a href="https://twitter.com/tsmith">Ty Smith</a></h4>
<p><a href="https://www.youtube.com/watch?v=5C5bgY84WXw">video from another conference, but it is the same content</a></p>
<p><em>Ty</em> already implemented integrations for several companies (Evernote, Twitter and now Uber). He gave a neat
talk how to utilize <code class="highlighter-rouge">content providers</code> and <code class="highlighter-rouge">intents</code> to communicate from one app to another. So if you have to do something similar, check out his talk!</p>
<p><br /></p>
<h4 id="suggestions-for-a-totally-better-programming-language---chet-haase-and-romain-guy">“Suggestions for a Totally Better Programming Language” - <a href="https://twitter.com/chethaase">Chet Haase</a> and <a href="https://twitter.com/romainguy">Romain Guy</a></h4>
<p><a href="https://academy.realm.io/posts/360-andev-2017-chet-haase-romain-guy-totally-better-programming-language/">video</a></p>
<p>I am not sure what I liked most, the <em>Emoji</em> based language features (see picture below) or to introduce some <em>politeness to the compiler</em>
like <code class="highlighter-rouge">doWork().please()</code> :-) - most funny talk I’ve joined so far in my life!</p>
<center>
<picture>
<img src="/images/blog/andev/emoji.jpg" alt="Suggestions for a Totally Better Programming Language" />
</picture>
</center>
<p><br /></p>
<h2 id="summary">Summary</h2>
<p>To make it short: <strong>Awesome conference!</strong> Great opportunity to meet the Android community and a big part of the Google crowd.
If you have the chance to visit the <em>360|Andev</em> - go for it!</p>
<center>
<picture>
<a href="http://upday.github.io/jobs/"><img src="/images/jobs/we-are-hiring.png" alt="Jobs at upday" /></a>
</picture>
</center>
<p><a href="https://upday.github.io/blog/360_andev/">360|Andev - Denver 2017</a> was originally published by upday dev at <a href="https://upday.github.io">upday tech blog - now at medium</a> on August 17, 2017.</p>
https://upday.github.io/blog/dealing-with-murphy-s-law-at-upday2017-07-24T04:39:55+00:002017-07-24T04:39:55+00:00Nicola Miottohttps://upday.github.ionicola@upday.com
<p>In this blog post we will provide a few insights on the architecture of the <em>upday</em> news app (both backend and frontend), focusing on the work that has been put on the resiliency and robustness of the components around the <strong>My News</strong> feature.</p>
<h2 id="upday--samsung">upday & Samsung</h2>
<p>Before digging into the technicalities, it’s worth to spend a few words on what <em>upday</em> is and which responsabilities we have towards our stake holders.</p>
<h3 id="the-app">The App</h3>
<p><em>upday</em> is a fruit of a collaboration between <a href="https://en.wikipedia.org/wiki/Axel_Springer_SE">Axel Springer</a> and <a href="https://en.wikipedia.org/wiki/Samsung">Samsung</a>. The end result is a news application for Android, pre-installed as the default news-app on most of the Samsung devices (S7, S8, A series etc).<br /></p>
<p>The one purpose of the application is to deliver relevant news content to the users. The goal is achieved by means of the two news streams that the app provides:</p>
<ul>
<li><strong>Top News</strong>: this section is an editorial curated news stream, whose titles and headlines are manually crafted by a team of editors (in each of the available countries). It’s supposed to provide the user with all the relevant news of the moment, independently from taste and preferences.</li>
<li><strong>My News</strong>: this is the “machine curated” news stream, generated by a swarm of super intelligent cyber beings enslaved to our will. Or at least that’s how business sometimes likes to sell it. The important thing is that this stream is personalized based on the user explicit feedback and settings and behaviour.</li>
</ul>
<h3 id="sla-with-samsung">SLA With Samsung</h3>
<p>Collaborations with big players come always at a cost. For the engineering team at <em>upday</em> this cost is the Service Level Agreement that we need to commit to.
So, when a problem with the service occurs, it’s categorized in a specific severity level that also defines then the actions to take. These are, in Layman’s terms, the 4 severity levels of our SLA:</p>
<ul>
<li><strong>Severity 4</strong>: An Error affecting non- <em>upday</em> ‘s Services with little or no Users impact. More concisely: no one cares. It’s hit once a single third-party picture wouldn’t load, for instance.</li>
<li><strong>Severity 3</strong>: Errors that causes degradation of the UX. Like when you receive a news about the next Justin Bieber concert although you explicitely stated that you like music.</li>
<li><strong>Severity 2</strong>: Partial unavailability of the services. Reached, among the other cases, when Top News stop being delivered in one country. Editorial laziness could be a cause.</li>
<li><strong>Severity 1</strong>: Full outage, across all features, across all countries. It’s the blue screen of death of <em>upday</em>.</li>
</ul>
<p>Each severity layer requires different response times, from days (less severe) to minutes (most severe). That’s why at <em>upday</em> we try <strong>not to ever exceed Severity 3</strong>, for the sake of our sleep and nerves.</p>
<h2 id="general-architecture-for-my-news">General Architecture For My News</h2>
<p>Talking about our backend technology, we have always tried to structure our architecture in a Micro-Service oriented way. Sometimes we have services that are actually small and with contained use-cases, sometimes we build the Goldman Sachs of backends. The important thing is that each team has its own responsabilities and takes care of its own sh…tuff.<br />
Therefore, each feature in the app has its own service or set of services, including Top New and My News. The outage of one does not imply the outage of the others.</p>
<p>Given that the focus of this article is the availability of My News (the most tricky maybe), we will start by giving a brief overview of the stream structure and the architecture involved in this feature.</p>
<h3 id="my-news-stream">My News Stream</h3>
<p>The My News section in <em>upday</em> is a potentially endless scrollable list of full screen cards, each displaying the content of a news. A sample:
<br />
<br />
<img width="300" style="margin-left:25%" src="/images/blog/upday_vs_murphy/mynews_card.png" />
<br />
<br />
Using our terminology, a card is called a note and a whole stream of notes symphony. Each note has constraints about how to select an article for the symphony, so it basically defines a subset of all existing articles. A couple of examples: a note could be just about fresh main stream news, another note could identify all trending news.</p>
<p>Each note is built differently, some of them are even provided by specific micro-or-so-services. The whole My News stream is then, to some extent, very modular. Some notes are also not constrained by the user preferences (to provide a more serendipitous experience).</p>
<h3 id="my-news-components">My News Components</h3>
<p>Below a simplified version of the components involved in the delivery of My News
<br /></p>
<p><img src="/images/blog/upday_vs_murphy/mynews_arch.jpg?2" alt="My News architecture" title="My News Architecute" /></p>
<p>Quick overview of the responsabilities:</p>
<ul>
<li><strong>CDN</strong>: edge cache</li>
<li><strong>API Gateway</strong>: performs authentication, SSL termination… and, most importantly, forwards each request to the right service. My News requests, specifically, are forwarded to the so-called <em>My news Request Proxy</em></li>
<li><strong>My News Request Proxy</strong>: exposes the public APIs and deals with the app needs</li>
<li><strong>Content Machine</strong>: the core of the news stream creation. It leverages Elasticsearch to create some of the notes, Preference provider to retrieve user preferences and some other services to build specific special notes</li>
<li><strong>Preference Provider</strong>: it’s responsible of providing user defined preferences and other inferred features, helped by other data mining components (not shown for simplicity)</li>
<li><strong>Special Notes</strong>: notes that require specific technology to be created. For instance collaborative filtering.</li>
<li><strong>App</strong>: take a guess</li>
</ul>
<p>Our <a href="https://www.infoq.com/news/2016/07/Design-Continuous-Evolution?utm_source=news_about_immutable_infrastructure&utm_medium=link&utm_campaign=immutable_infrastructure">infrastructure is immutable</a> and <a href="https://aws.amazon.com/autoscaling/">each component is autoscaled</a>, so we will not go into the details of what happens if a developer shuts down a single instance (answer: nothing).</p>
<h3 id="the-problem">The Problem</h3>
<p>Each of these components could fail and will fail at some point. The challenge is to make sure that the user doesn’t notice or doesn’t get really annoyed.</p>
<h1 id="unleash-the-chaos-monkey1">Unleash The <a href="https://github.com/Netflix/chaosmonkey">Chaos Monkey</a><sup><a href="#footnote-1">1</a></sup></h1>
<p>We will see now what the failover mechanism in place for each of the surprises that <a href="https://en.wikipedia.org/wiki/Murphy%27s_law">Murphy</a> has prepared for us.</p>
<h2 id="special-notes">Special Notes</h2>
<p>As mentioned already, special notes define article subsets that are computed using specific technology. Our collaborative filtering note is, for instance, generated leveraging Apache Mahout. It happens that these components slow down or break once in a while, given their computationally intensive tasks and dependencies.
<img style="margin: auto; margin-left: 25%; margin-top: 10px;" src="/images/blog/upday_vs_murphy/chaos_special.jpg?s" /><br /></p>
<p><strong>Mitigation</strong>:</p>
<ul>
<li><a href="https://martinfowler.com/bliki/CircuitBreaker.html">Circuit breakers</a>: the content machine will give these components a maximum amount of time to return content. Once over, the note will simply be skipped. Given that thes notes are less than 10% of the whole symphony, the user will not notice any difference in the app, although it might not get highly converting content for a while.</li>
</ul>
<p><strong>Severity level: 4</strong></p>
<h2 id="elasticsearch-cluster">Elasticsearch Cluster</h2>
<p>Although it’s quite a tough challenge to take down an Elasticsearch cluster, we still managed quite a few times to do it. AWS once had networking issues, hence the whole cluster has been partitioned away from the Content Machine. This basically means that the Content Machine is isolated from our content storage.<br />
Mitigation:
<img style="float: right; margin-top: 30px; margin-left: 10px; width: 350px" src="/images/blog/upday_vs_murphy/chaos_es.jpg?s" /><br /></p>
<ul>
<li><strong>In memory cache, layer 1</strong>: when content for a note is required, a bigger amount is fetched from Elasticsearch before continuing with the stream generation. From the whole pool of articles that matches the note criterias, only the needed ones are used. Essentially, other instances of the same notes are picked from the in memory pools.</li>
<li><strong>In memory cache, layer 2</strong>: a bunch of periodic background jobs are collecting (from Elasticsearch) content for preference independent notes. These notes are then generated purely out of memory when the user requests a stream.</li>
<li><strong>In memory cache, layer 3</strong>: a bunch of predefined symphonies is kept in memory as a final fallback, in case all the pools have been fully used or expired.</li>
</ul>
<p><strong>Severity level: 3-4</strong> (for at least 30 minutes. Degradation might become more noticeable after a longer time)</p>
<h2 id="preference-provider">Preference Provider</h2>
<p>By being an “online synchronous component” (the load is proportional to the traffic), sudden traffic spikes and slow autoscaling can make temporarily shake temporarily, with slow downs and unavailabilities.<br />
<img style="margin: auto; margin-left: 25%; margin-top: 10px;" src="/images/blog/upday_vs_murphy/chaos_pp.jpg" /><br /></p>
<p><strong>Mitigation</strong>:</p>
<ul>
<li><strong>Circuit breaker and content degradation</strong>: the content machine will again open the circuit if the Preference Provider takes longer then a minimum amount of time to reply or becomes unavailable. In this case, a symphony with all preferences switched on is generated.</li>
</ul>
<p><strong>Severity level: 3</strong></p>
<h2 id="rabbit-mq">Rabbit MQ</h2>
<p>The <a href="https://www.rabbitmq.com/">RabbitMQ</a> cluster has been configured for high availability. Yet, it happened that a misconfiguration in the maximum open files on conjunction with a huge traffic slash, made the cluster partially go down and brain split during a wrong recovery (worst weekend ever for the the guys on call).<br />
<img style="margin: auto; margin-left: 25%; margin-top: 10px;" src="/images/blog/upday_vs_murphy/chaos_rabbit.jpg" /><br /></p>
<p><strong>Mitigation</strong>:</p>
<ul>
<li><strong>Switch to synchronous</strong>: The My New Request proxy, upon recognizing the unavailability of the message queue service, will start calling the synchronous APIs of the Content Machine. The solution always gave us enough time to address the issue.</li>
</ul>
<p><strong>Severity level: None</strong> (although the Content Machine might suffer under prolongued exposure to synchronous requests).</p>
<h2 id="content-machine">Content Machine</h2>
<p>This component usually never fails, but when it fails, it fails epicly. It’s usually when the on call engineer start getting cold sweats.<br />
<img style="margin: auto; margin-left: 25%; margin-top: 10px;" src="/images/blog/upday_vs_murphy/chaos_cm.jpg" /><br /></p>
<p><strong>Mitigation</strong>:</p>
<ul>
<li><strong>Default news</strong>: the My News Request proxy keeps a cache of up-to-date default symphonies (fetched periodically from the content machine itself), that can be used during a prolongued outage of the content machine. These symphonies are not personalized.</li>
</ul>
<p><strong>Severity level: 3</strong></p>
<h2 id="full-failure">Full Failure</h2>
<p>Sometimes the more external components go down (API-Gateway, My News proxy) or maybe there are even prolongued failures of multiple internal services. In these cases, there is not much to do if not praying…
<img style="margin: auto; margin-left: 25%; margin-top: 10px; margin-bottom: 10px;" src="/images/blog/upday_vs_murphy/chaos_all.jpg" /><br />
… it’s not true. It’s in these cases that the mother of all failovers kicks in. In order to give time to the ops to figure out the issue and fix it, the whole infrastructure is basically replaced. How? The solution below:
<img width="390px;" style="margin: auto; margin-left: 25%; margin-top: 10px; margin-bottom: 10px;" src="/images/blog/upday_vs_murphy/failover.jpg" /><br />
The A record of the DNS pointing to the origin (namely the API Gateway) is replaced with the IP of <strong>The Failover</strong>.
This component has been built keeping the complexity as low as possible and loaded with monitors and alarms. Its only purpose is to fetch a bunch of symphonies periodically to be delivered <strong>as they are</strong> upon request. This usually gives us around 30 minutes to react and try to fix the issue, before most of the users start calling customer support.
<br />
<br />
<strong>Severity level: 3</strong> (for around 30 minutes)
<br /><br />
So far we never went beyond Severity level 3.</p>
<h1 id="conclusion">Conclusion</h1>
<p>This is just an overview of how we partly manage our backend availability at <em>upday</em>. These are not the only measures that we use: many monitors and alerts (with <a href="https://www.datadoghq.com/">Datadog</a> btw) have been deployed, some of them give us early clues of potential future problems (for instance low heap space on some instances, load going above average…). This gives us the chance to address issues before they explode into minor or major outages.<br />
On top of this, it’s worth mentioning the comprehensive amount of unit, integration and system tests that must be developed before any change can be deployed to production. Trust me: there would be a few outages per week if this would not be done. We are not in a condition where we can “simply hot fix production”.<br />
This is also just the current stage of our system: as we started, we had almost nothing of this in place and failure by failure we became more aware of how careful one has to be when dealing with big stakeholders and micro services (pun intended). Now we can kind of sleep well… usually.</p>
<p>PS: bring it on, Murphy.</p>
<p><br />
<a name="footnote-1">1</a>: We didn’t actually use Netflix Chaos Monkey, but we developed a home-made solution by following their example.</p>
<p><a href="https://upday.github.io/blog/dealing-with-murphy-s-law-at-upday/">Dealing With Murphy's Law at upday.</a> was originally published by upday dev at <a href="https://upday.github.io">upday tech blog - now at medium</a> on July 24, 2017.</p>
https://upday.github.io/blog/upday_blogathon2017-07-20T05:00:00+00:002017-07-20T05:00:00+00:00Robert Bordohttps://upday.github.iorobert@upday.com
<left>
<picture>
<img src="/images/blog/blogathon_2017/blogathon_2017_logo.png" width="10%" height="10%" alt="Blogathon 2017 Logo" />
</picture>
</left>
<h2 id="what-and-why">What and why?</h2>
<p>Here at upday we have a lot of talented engineers doing very interesting “stuff” and using up-to-date (some would even say top-notch) technologies.
<strong>Doing good is fine, but talking about it is even better.</strong> The problem is that not every engineer likes writing about his profession.
Often I hear: <em>“this wouldn’t be interesting to anyone else”, “there are already articles about it”, “I don’t have time for this” or “I simply cannot write”.</em>
All this might be true. But for me, it’s not enough of an excuse.</p>
<p>Last year we started our <a href="http://upday.github.io/">upday tech blog</a> to attract more talented people and of course,
to get attention in the tech community. This worked out quite well. We received promising applications,
a few of us got invited as speakers to international conferences and we also organised well-received meetups.
Unfortunately, the driving force behind most of these activities left the company, hired by a much bigger player
(another proof our efforts bear fruit). And Since then our blog has been more or less inactive.</p>
<p>This needed to change. <strong>But how can one convince people to invest time and energy into something they don’t really like or want to do?</strong></p>
<h2 id="how">How?</h2>
<p>At upday, from time to time we organise <strong>Hackathons</strong>. During a Hackathon, people work intensely in unusual team formations
(often business people also participate), test new technologies and dive into certain topics in order to produce something valuable,
interesting or surprising within only a few hours of work. A few of these results have actually found their way into our product.
Others were just cool. And some, of course, just sucked.</p>
<p>Our plan now was to “abuse” one of those Hackathons to generate blog posts. The idea of a <strong>Blogathon</strong> was born.
Okay, I know what you might be thinking right now.</p>
<p>Anyway, presenting this idea provoked mixed reactions. Everything from “cool” to “it sucks big time”.
But no risk no fun, it’s only painful at the start and attending was optional anyway.</p>
<h3 id="timetable">Timetable</h3>
<ul>
<li>we announced the Blogathon two weeks in advance</li>
<li>one week before the Blogathon, we set up an <strong>Ideas Board</strong> that served two purposes:
<ul>
<li>to let people start thinking about possible topics</li>
<li>to collect topics and trigger the creativity of others</li>
</ul>
</li>
<li>we had the Blogathon on a Thursday</li>
</ul>
<left>
<picture>
<img src="/images/blog/blogathon_2017/blogathon_2017_ideas_board.jpg" width="50%" height="50%" alt="Ideas Board" />
</picture>
</left>
<h3 id="agenda">Agenda</h3>
<left>
<picture>
<img src="/images/blog/blogathon_2017/blogathon_2017_agenda.jpg" width="50%" height="50%" alt="Agenda" />
</picture>
</left>
<h2 id="and">And…?</h2>
<p>To make a long story short, our Blogathon was quite a success. <strong>We ended up with 8 finished or nearly finished articles</strong> covering topics
from tutorials to personal insights from our daily business. You will be able to read the best of them in our <a href="http://upday.github.io/">upday tech blog</a>
during the next few weeks.</p>
<p>To finally <strong>publish</strong> them, all articles have to pass a <strong>quality gate</strong>.
We are used to challenging ourselves in terms of high standards so why shouldn’t this apply to our written output as well?</p>
<p>These are the finishing steps to get a blog post out on the street:</p>
<ul>
<li><strong>finish</strong> your article to the point you are happy and proud of it</li>
<li>let one or two <strong>peers review</strong> it for clarity (this might be an iterative process)</li>
<li>let a <strong>native speaker review</strong> it</li>
<li>clone and create a branch of our blog repository</li>
<li>add your article (Markdown)</li>
<li>optimise the <strong>layout</strong>, give it a <strong>readable structure</strong>, <strong>emphasise</strong> important passages and add <strong>keywords</strong></li>
<li>give your article a <strong>final review</strong></li>
<li>merge the branch and publish it</li>
</ul>
<h2 id="conclusion">Conclusion</h2>
<p><strong>Everybody can write</strong> and everybody should! Sharing thoughts or findings with others is rewarding.
It is more blessed to give than to receive. Writing also serves as a good reflection on what you have achieved.
It might also inspire others or merely put a smile on someone else’s face. Of course it requires work and time but it is also fun and satisfactory.
It is not unlikely that you will actually help someone else with exactly the piece of information he or she were desperately looking for.</p>
<p><strong>For us this Blogathon was a success.</strong> As I mentioned before, only in the beginning it hurt, but just a little bit.
I’m convinced we are now motivated enough to produce blog posts more regularly and with the knowledge that <em>“I’m able to do it
because I already did”</em> and <a href="http://upday.github.io/blog/tech-talks/">“You Do Have Something To Say!”</a> gave a lot of drive and confidence to everyone involved.</p>
<p>We will have another Blogathon for sure sometime.</p>
<center>
<picture>
<a href="http://upday.github.io/jobs/"><img src="/images/jobs/we-are-hiring.png" alt="Jobs at upday" /></a>
</picture>
</center>
<p><a href="https://upday.github.io/blog/upday_blogathon/">upday Blogathon 2017</a> was originally published by upday dev at <a href="https://upday.github.io">upday tech blog - now at medium</a> on July 20, 2017.</p>
Part 1]]>https://upday.github.io/blog/holacrcy-part-12017-06-26T19:39:55+00:002017-06-26T19:39:55+00:00Peter Kraußhttps://upday.github.iopeter.krauss@upday.com
<p><strong>What is the right approach to being as happy, productive and as innovative as possible?</strong> Well, I guess this question
pops up in many companies and there are many books out there trying to give an answer to this question.
However, I deeply believe <strong>there is not one answer “to bind them all”</strong>.</p>
<p>However some <strong>principles</strong> are helpful
<strong>to create a healthy environment</strong> - this means for us <a href="http://upday.github.io/about/">updudes</a>:</p>
<ul>
<li>you need independence to make your own decisions</li>
<li>you need to know what you are responsible for</li>
<li>you need to know what you can expect from others and what they are responsible/accountable for</li>
</ul>
<p>If all these conditions are met, the probability you finish your task with the highest quality and efficiency is high - and you may even have some fun in the meantime.</p>
<p>One guide that everyone is talking about in the moment is <strong>Holacracy</strong>. So we decided to have a closer look.
And what did we realize?</p>
<p><strong>It is not the “Silver Bullet” to solve all our problems!</strong> <em>(What a surprise!)</em></p>
<p>But <strong>Holacracy offers many tools</strong> we started using to come closer to our <em>Garden of Eden</em>.
In the next sections I will describe some key aspects of Holacracy. Our experiences will be part of a follow-up post.</p>
<h2 id="the-cto-is-dead---long-live-the-constitution">The CTO is Dead - Long Live The Constitution</h2>
<p><strong>The classical hierarchical system</strong> - one person is in charge and takes all the decisions -
<strong>worked pretty well for a few hundred years.</strong> It assumes this person is able to take the best possible decisions
because he or she foresees every requirement and fully understands the big picture. This might be true
for assembly lines producing simple things. But it already becomes unlikely if you think about fully
automated, robot-controlled assembly lines in car production. <strong>It is completely impossible in an agile and fast
changing software development process</strong>.</p>
<p><strong>What could be the alternative?</strong> Anarchy, grass-roots democracy based decisions or maybe hiring an almighty genius who
knows everything? But this would be the omniscient leader again… And how would you know he/she knows everything?</p>
<p><strong>The answer Holacracy provides is an accurate set of rules</strong> to shift the power to a constitution
that is mandatory for every member of the organization. As in every other constitutional driven
culture, <strong>you are allowed to take your own decisions as long as you stick to the given rules</strong>.</p>
<p>The provoking title of this section could suggest <strong>Holacracy is not a hierarchical system</strong> - far
from it! There is still someone (or even a group of people) that takes decisions for the whole company. Like a government
may decide to build an airport, but all the technical decisions are taken by someone
who actually knows how to construct a runway or a hangar (maybe a bad example for a Berlin based company 😃).</p>
<p><strong>The main purpose of the constitution is to provide Roles</strong> <em>(who is accountable for what)</em> <strong>and Governance Processes</strong>
<em>(fix or improve the roles)</em>.</p>
<h2 id="role-vs-soul-vs-circle">Role vs. Soul vs. Circle</h2>
<p>The <strong>definition of a Role</strong> consists of three elements:</p>
<ul>
<li>a <strong>Purpose</strong></li>
<li>a <strong>Domain</strong> it controls</li>
<li>a set of <strong>Accountabilities</strong> and <strong>corresponding Domains</strong></li>
</ul>
<p><strong>Only the Purpose is mandatory</strong>, the Accountabilities and Domains can evolve over time.</p>
<hr />
<p>As an example we have a look at the <a href="https://app.glassfrog.com/roles/1787">Developer Role from GlassFrog</a> -
which is a web based tool developed by <a href="http://www.holacracy.org/">HolacrcyOne</a>.</p>
<p><strong>Developer</strong></p>
<p><em>Purpose:</em></p>
<p>Satisfy the customer, with Product Vision as proxy, through early and continuous delivery of truly valuable software.</p>
<p><em>Accountabilities:</em></p>
<p>Implementing GlassFrog user stories, bug fixes, and chores in alignment with:</p>
<ul>
<li>development practices set by Scrum Master</li>
<li>implementation standards set by Quality Guardian</li>
<li>testing new functionality before each production deployment</li>
<li>providing input into estimation at the request of Scrum Master</li>
<li>sharing development challenges with Product Vision, API Product Designer, and Developer as challenges arise</li>
<li>demonstrating working software to Product Vision regularly and frequently, and integrating feedback surfaced by Product Vision</li>
</ul>
<hr />
<p><strong>A definition like this makes it explicit what you can expect</strong> from a developer and on the other hand a dev
knows what he/she is accountable for.</p>
<p><strong>There is a big difference between the people</strong> that are working in
an organization <strong>and the roles they hold</strong>. Decoupling these two elements allows for drastic improvements to an
organization’s culture.</p>
<p><strong>Conflicts in organizations</strong> are mostly <strong>based on disagreements between different roles</strong> involved.
But often they are mistaken as clashes between the people holding these roles. This makes the
conflicts personal and we try to resolve them by smoothing the personal dispute rather than resolving the underlying
issues in the organizational structure.</p>
<p>For example, if I as a developer need time to
unit test a feature, this may clash with a product owner, who wants that particluar feature out as soon as possible.
It seems that he/she is not trusting your judgement.</p>
<p><strong>But in fact the conflict is not personal at all.</strong>
Both of your are just filling your role and there is a tension that can be used to improve the organization
even further.</p>
<p><a href="http://www.holacracy.org/">HolacrcyOne</a> provides a nice video that explains what a Tension can be:</p>
<h2 id="what-is-a-tension">What is a Tension?</h2>
<div class="yt-video"><iframe width="560" height="315" src="https://www.youtube.com/embed/MUHfVoQUj54" frameborder="0" allowfullscreen=""></iframe></div>
<p><br /></p>
<p>How do the <strong>Circles</strong> fit into the picture now? Well, actually <strong>a Circle is just an accumulation of some
Roles for a broader Purpose</strong>. So basically you can say, when a Role becomes too much - it becomes a Circle. It also follows
the same Rules - it needs a Purpose and a Domain it controls. In other words: <strong>A Circle has the autonomy and authority to
self-organize all Roles within the Circle.</strong></p>
<p>This self-organization follows strict rules, called <strong>Governance</strong>, which will be explained in the
next post.</p>
<h2 id="tldr">tl;dr</h2>
<p><strong>Holacracy offers a comprehensive set of Rules</strong> to move away from a top-down hierarchy to a <strong>constitutional based organization</strong>
that consists of <strong>Roles and Circles</strong>. The objective of Holacracy is <strong>to identify so called Tensions</strong> and harvest their energy
<strong>to improve the organization constantly</strong>.</p>
<p><a href="https://upday.github.io/blog/holacrcy-part-1/">Holacracy - Whatˋs The Fuss About? <br> Part 1</a> was originally published by upday dev at <a href="https://upday.github.io">upday tech blog - now at medium</a> on June 26, 2017.</p>
https://upday.github.io/unpublished/agile2017-04-23T11:00:00+00:002017-04-23T11:00:00+00:00upday devhttps://upday.github.iodev@upday.com
<h2 id="about-the-role">About the Role</h2>
<p>We are looking for an Agile Coach to coach, enable and empower our teams. You will collaborate with
cross functioning, highly skilled engineering teams in a fast moving environment and be a part of
creating a product that combines the latest technology with great journalism in an unique way.</p>
<p><strong>What you will do:</strong></p>
<ul>
<li>Contribute to building an environment where continuous improvement and learning are in focus and where everyone’s common goal is to deliver outstanding software as fast as possible while keeping quality high</li>
<li>Support the teams to face tough challenges and to find improved ways of collaborating</li>
<li>Ensure transparency and visibility on all dimensions</li>
<li>Actively identify areas of improvement and conceptualize methods of how to work more effectively</li>
<li>Work from our office in Berlin</li>
</ul>
<p><strong>Who you are:</strong></p>
<ul>
<li>You have deep knowledge about and experience of different agile practices, and know what is useful to try depending on the situation</li>
<li>You like to get your hands dirty. This means that you enjoy hands-on coaching of teams, as well as of other agile coaches, developers, product owners, CTO and executives</li>
<li>You are not afraid to raise issues and you see impediments as opportunities to change</li>
<li>You use your skills in facilitation, communication and group dynamics to help people and teams grow</li>
<li>You inspire others and are good at what you are doing, but are always curious and feel that you have much left to learn</li>
<li>You are a true servant leader and you find satisfaction in supporting and seeing others succeed while your own deliverables are often intangible</li>
<li>You lead by example and take great pride in your work</li>
</ul>
<p><strong>This would be a plus:</strong></p>
<ul>
<li>You have been developing software yourself as an engineer</li>
<li>You have experience in scaling one team to many teams and handle agility in growth</li>
</ul>
<p>If this is you, <a href="mailto:jobs@updudes.net">get in touch with us and send us an email to jobs@updudes.net</a>!</p>
<p><a href="https://upday.github.io/unpublished/agile/">Agile Coach</a> was originally published by upday dev at <a href="https://upday.github.io">upday tech blog - now at medium</a> on April 23, 2017.</p>
https://upday.github.io/unpublished/hr-intern2017-04-23T11:00:00+00:002017-04-23T11:00:00+00:00upday devhttps://upday.github.iodev@upday.com
<h2 id="about-the-role">About the Role</h2>
<p>You are a psychology student and have a passion for IT and all the nerdy guys around it?
We are looking for someone who helps us to increase our team and find the best developers in the world. Your role would be to identify and get in touch with them. <br />
This includes the preparation of our own meet-up, search for conferences and come up with new ways to make upday even more public in the Berlin tech scene.</p>
<p><strong>You are:</strong></p>
<ul>
<li>A master student in Psychology looking for an internship</li>
<li>Seeing problems yourself and take ownership solving them</li>
<li>Specialized in Work and Organizational Psychology</li>
<li>Interested in Software development</li>
<li>Striving to develop yourself and your team</li>
<li>Cool to be around</li>
</ul>
<p><strong>Requirements:</strong></p>
<ul>
<li>fluent english</li>
<li>You are a team player with experience working in an agile environment</li>
<li>Willingness to develop your skills</li>
<li>Understanding that professionalism is not the absence of fun and that absence of fun is unprofessional</li>
<li>An interesting hobby, like brewing beer or visiting Mars</li>
</ul>
<p>If this is you, <a href="mailto:jobs@updudes.net">get in touch with us and send us an email to jobs@updudes.net</a>!</p>
<p><a href="https://upday.github.io/unpublished/hr-intern/">Internship - Human Relations</a> was originally published by upday dev at <a href="https://upday.github.io">upday tech blog - now at medium</a> on April 23, 2017.</p>
https://upday.github.io/blog/public_speaking_intros2017-03-14T04:39:55+00:002017-03-14T04:39:55+00:00Florina Muntenescuhttps://upday.github.ioflorina@upday.com
<p>Public speaking pushed me out of a lot of my comfort zones. I got to learn not only about the content I was presenting and about the delivery, but also about myself. I found out how I react in certain situations, what feels genuine and what doesn’t, and what I should work on more.
<br /></p>
<blockquote>
<p><em>Public speaking is a journey of discovering yourself.</em></p>
</blockquote>
<p><br />
For example, I learned that I can control my speech rate and can use that to improve the effect of my words. But most of all, I learned that indeed, public speaking is one of humanity’s biggest fears, but that I can restrain it and decrease it and that hopefully, one day, I can transform it into contagious energy and enthusiasm for my audience. </p>
<p>I saw that before my talk, doing breathing exercises and a power pose stabilises my heart rate and reduces my nervousness. But, once I start talking, I still need a few minutes until I’m calm, and get into my speaking flow. My voice still shakes a bit.</p>
<p>The <a href="https://www.ted.com/read/ted-talks-the-official-ted-guide-to-public-speaking" target="blank">public speaking theory</a> says that the intro and the conclusion should be the most powerful parts of a talk. I’m far from mastering the conclusion, but I’ve been working on the intro. </p>
<p>I decided to embrace the emotions and share them with my audience, to use them as a way of creating a better connection with them.</p>
<p>The theme of my latest talk was <em>“Telling Our Story”</em>. I started by telling the audience that it’s hard for me to share a personal story in front of strangers and that I want to get to know them a bit first. Then I introduced myself and asked who else in the audience is like me, a developer, who is a QA, who’s a Product Manager and so on, trying to find out who my audience is. Once I knew them a bit, so we weren’t strangers anymore, I declared that I was ready to start sharing my story.</p>
<h4 id="this-intro-served-several-purposes">This intro served several purposes:</h4>
<ul>
<li><strong>It allows the audience to identify with the speaker</strong> — for most of us is hard to share personal details on stage.</li>
<li>It allows me, the speaker, to <strong>get to know the audience</strong> and then, later on, in the intro but also in the conclusion, to mention that what I’m saying is addressed to them; to my fellow devs, to the QAs and PMs in the room. Like this, I’m trying to make the communication feel special for every one of them. To express the fact that <strong>each of them is important to me and are the target of my message</strong>.</li>
<li>It gives me those two minutes that I need for my voice to <strong>calm down</strong>. </li>
</ul>
<p>Of course, the intros differ from one talk to another, depending on the topic, and the target audience. But the main roles for me are to create a connection with the audience and to give me those few minutes to get into my flow. </p>
<p>It took me quite a few talks with crooked beginnings to find what works for me. So don’t be afraid of talking, of experimenting and finding out what works best for you and what doesn’t because being genuine is important.</p>
<hr />
<p>How are your intros? How do you deal with your initial nervousness? How do you try to connect with the audience? I would love to learn more from your experiences.</p>
<center>
<picture>
<a href="http://upday.github.io/jobs/"><img src="/images/jobs/we-are-hiring.png" alt="Jobs at upday" /></a>
</picture>
</center>
<p><a href="https://upday.github.io/blog/public_speaking_intros/">About Intros, Nerves and Discovering Yourself</a> was originally published by upday dev at <a href="https://upday.github.io">upday tech blog - now at medium</a> on March 14, 2017.</p>
https://upday.github.io/blog/public_speaking_blank_slides2017-03-08T04:00:00+00:002017-03-08T04:00:00+00:00Florina Muntenescuhttps://upday.github.ioflorina@upday.com
<p>My talk at the <a href="https://www.womentechmakers.com/iwd17" target="blank">International Women’s Day event</a>, organised by <a href="https://www.womentechmakers.com/" target="blank">Women Techmakers</a> gave me the chance to share my story about public speaking and also to try something new: using <strong>blank slides</strong>.</p>
<center>
<picture>
<a href="/images/blog/public_speaking_blank_slides/google_slides.png"><img src="/images/blog/public_speaking_blank_slides/google_slides.png" alt="Public Speaking Blank Slides" /></a>
<figcaption>Screenshot from my Google Slides</figcaption>
</picture>
</center>
<p>The time allocated for my talk was 25min + 5min for Q&A. Since this was a non-tech talk that didn’t require much visual support, I decided to combine text slides with blank slides. You can find the final deck <a href="https://www.slideshare.net/FlorinaMuntenescu/tech-talks-you-do-have-something-to-say-why-and-how-to-start" target="blank">here</a>.</p>
<h4 id="my-strategy-was-this">My strategy was this:</h4>
<ul>
<li>Use <strong>blank slides</strong> for the moments when I’m telling <strong>my story</strong>. Like this, the focus of the audience is on what I’m <em>saying</em>.</li>
<li>Use <strong>text slides</strong> only for <strong>quotes</strong> and <strong>main take-aways</strong>. These would only appear in the key moments of my presentation and the audience should focus on them. The effect of the take-aways is now amplified, since these are the only parts that the audience both <em>hears and sees</em>.</li>
</ul>
<p>Usually in my tech talks, I tend to have one slide per idea. So I kept the same approach now, even if my slides were blank. This also gives me a sign of when I should have a small break in my speaking flow.</p>
<h4 id="there-are-a-few-possible-downsides">There are a few possible downsides:</h4>
<ul>
<li>The <strong>IT audience</strong> is used to talks where they always have a visual guidance, so having a black screen all of the sudden can be a surprise. I saw this effect when rehearsing my talk with a friend. But, after the initial <strong>surprise</strong>, I could see that my strategy is working: he was only looking at the slides when there was something there and the rest of the time, his attention was on what I was saying.</li>
<li>If the audience gets distracted by their phone when they’re trying to pay attention to the talk again, it will be harder to grasp the context because of the lack of visual aid. To diminish this effect, the time between text slides was only a few minutes.</li>
</ul>
<h2 id="talk-time-how-did-thingsgo">Talk time! How did things go?</h2>
<h4 id="from-a-speakers-perspective">From a speaker’s perspective:</h4>
<ul>
<li>Given that the slides are blank, I had <strong>no more support for what’s coming next</strong> in my talk. I had to rely either on the speaker’s notes or on knowing the talk really well.</li>
<li>The tendency of looking at the slides and turning the back to the audience decreased so the <strong>communication with the audience improved</strong>.</li>
<li>I could see that the audience was looking at me more and at the slides only when there was something there, which was expected. People didn’t seem to get distracted that much, which was great!</li>
<li>The importance of the body language, tone of voice and speech pace increased. So I tried to use all of these to give more meaning and power to my words and also to connect more with the audience.</li>
</ul>
<h4 id="from-the-audiences-perspective">From the audience’s perspective:</h4>
<ul>
<li>Like my rehearsal audience, they were <strong>surprised</strong> the first time when they saw the blank slide. They thought that there was a technical issue and the slide is not loading.</li>
<li>After the first blank slide everything was ok.</li>
</ul>
<h2 id="blank-slidesyay-or-nay">Blank slides — yay or nay?</h2>
<h3 id="yay">Yay!</h3>
<ul>
<li>The slides with text or image become more powerful.</li>
<li>The speaker has a better control over the audiences’ attention, by removing the distraction that can be caused by slides.</li>
<li>Great in short or really engaging talks, where the audience doesn’t have the time to get sidetracked by phone/email/social media.</li>
</ul>
<h3 id="nay">Nay!</h3>
<ul>
<li>If the audience is not-used to blank slides, the first one will be a surprise and might disturb the audience. One possible solution would be to use the blank slide very early in the talk, maybe even from the first sentences.</li>
<li>Using blank slides means that you don’t have any support for what’s next in your talk. So, knowing the order of your ideas, becomes more important and the preparation time might get higher.</li>
</ul>
<p><br />
Have you ever used blank slides in a talk? I would love to know more about your experience, the problem faced and the solutions you found.</p>
<center>
<picture>
<a href="http://upday.github.io/jobs/"><img src="/images/jobs/we-are-hiring.png" alt="Jobs at upday" /></a>
</picture>
</center>
<p><a href="https://upday.github.io/blog/public_speaking_blank_slides/">A Public Speaking Experiment: Blank Slides</a> was originally published by upday dev at <a href="https://upday.github.io">upday tech blog - now at medium</a> on March 08, 2017.</p>
(or, why user feedback sucks)]]>https://upday.github.io/blog/product_management_user_feedback2017-03-08T04:00:00+00:002017-03-08T04:00:00+00:00Madeleine Wanthttps://upday.github.iomadeleine@upday.com
<p><em>“Until now, sociologists have depended on limited data sets and surveys that tell us how people say they think and behave, rather than what they actually do.”</em></p>
<p align="right">
- <i>(Pentland, 2015)</i>
</p>
<p>I grew up thinking I was clever. My grades seemed to say so, and my parents certainly did. But I wasn’t sure exactly what that meant, and at some point I settled on the conclusion that being clever means you probably don’t need to check your math. If only I’d verbalised that idea and been gently corrected early on, I would have saved myself years wasted assuming and spent them learning instead.</p>
<p>For several years now I’ve enjoyed a trajectory of increasing leadership with very little authority. As a Product Manager I lead a product – but I don’t have any formal technical qualifications. I also lead a team and a vision – but nobody reports to me. This is typical of the Product Manager role, and makes it challenging and rewarding at the same time. Strong Product Managers don’t have to bring specific technical skills to the team, but they do need <strong>soft skills</strong><em>, - “the ability to steer and navigate, intangible qualities and behaviors”</em> (Cheng, 2016). Everything that allows you to understand, communicate, build consensus, debate and direct is a soft skill, and Product Managers do these things every day.
It makes sense: why else would your team jump on-board for your vision? You have to be able to communicate it compellingly. Why should the business allocate resources to your plan? You need to be able to argue it persuasively. How can you be confident it will succeed? You need to listen to the people who will make it succeed – <strong>the users</strong>.</p>
<h2 id="product-management-then">Product Management Then</h2>
<p>Traditional definitions of the Product Manager/Product Owner (PM/PO) role have focused on the blend of skills it requires. Being a great PO includes being a customer delighter, storyteller, delegator, developer, knowledge broker, conflict resolver and escalator (Lindstrom). Others have visualised similar definitions in diagrams like this one <sup id="fnref:1"><a href="#fn:1" class="footnote">1</a></sup>:</p>
<center>
<picture>
<a href="/images/blog/product_management_user_feedback/fig1.png"><img src="/images/blog/product_management_user_feedback/fig1.png" alt="Product Management then" /></a>
</picture>
</center>
<p>Let’s call this <strong>Product Management Then</strong>. Why? It’s meant to illustrate product’s jack-of-all-trades (master of none) nature. A PO needs some Business Analyst skills; to be able to calculate the value (e.g. revenue, user acquisition, market positioning) of potential features given the cost of development. They should also be confident in UX design, including visualising concepts and wireframing. Creating user feedback loops has also been part of the PM’s role and seems to fall within UX. Finally, a PM should be technically literate enough to lead a team of engineers, forecast dependencies, manage development projects and facilitate the process of solution-storming. All this plus a little sprinkle of something extra (management calls it ‘X-Factor’, I call it working overtime) has thus far been the recipe for a PM.</p>
<p>But it’s no longer adequate. Technology has moved on and so has product. This venn diagram is as helpful to an aspiring PM as a map of the USSR is to a traveller (if you see this, Mr. Eriksson, I am a big fan of yours).</p>
<p>So what exactly is missing here? <strong>Data is</strong>. Fundamental data science practices are the new necessity of Product Management. With data we can know what we’ve never been able to know before, and we can gather detailed feedback from users in real time, without them ever having to say a word. Better yet, data is what humans are not: <strong>unbiased</strong>. Another side to the users’ story is now audible, and we need to hear it. To meet Lindstrom’s criteria of being customer delighters, storytellers and knowledge brokers, being data-illiterate is no longer acceptable.</p>
<h2 id="do-as-i-do-not-as-i-say">Do As I Do, Not As I Say</h2>
<p>When we ask users for feedback, we get responses - and that’s the problem. Traditional feedback methodologies like surveys, focus groups and NPS questionnaires invite the user to share their opinion and to consider, edit and package their response. <strong>They decide what to share and how to share it</strong>. This feedback is at best informational, and at worst useless – especially if it’s given in a public or direct interaction (focus groups, interviews) where a myriad of social dynamics affect what is shared. For example, people will often subdue their own judgement in order to agree with the consensus of a group, even when they truly disbelieve in the group’s decision – a cognitive error called ‘social proof’ (Dobelli, 2013). This kind of response manipulation simply doesn’t happen when the participant is in a solitary, non-social setting.</p>
<p>Highly fertile grounds for the exploration of <strong>truth via data</strong> are found in one of the world’s largest dating sites, OkCupid. Christian Rudder, founder of OkCupid and author of Dataclysm describes how data steps in where reported feedback falls short: <em>“unfortunately, surveys have historically been unable to uncover true attitudes on topics such as race, sexual behaviour… because respondents edit their answers”</em> (Rudder, 2015). Dataclysm starts with a stark example of this self-editorial. According to <em>‘Wooderson’s law’</em> <sup id="fnref:2"><a href="#fn:2" class="footnote">2</a></sup>, heterosexual men who sign up to OkCupid usually specify the age range of women they’re seeking to meet as 5-10 years younger or older than themselves. So a 30-year-old man states that he’s looking to meet a woman 20-35 years old, and as his age increases, so does the range of the women he is looking for. But the data tells a different story: the female OkCupid members that the men rate as most attractive are aged 20-23, no matter the age of the man. A 20-year-old man rates 20-year-old women as most attractive, and a 50-year-old man does too. There is <strong>no deviation</strong> from this pattern among any male age group. Rudder offers a plausible explanation: <em>“I don’t think that anyone is intentionally misleading us when they give OkCupid their preferences… I see this as a statement of what men imagine they’re supposed to desire, versus what they actually do. The gap between the two ideas just grows over the years”</em> (Rudder, 2015). In this case, the explicit feedback is only the tip of the iceberg.</p>
<p>I discovered a similar example recently, while comparing user feedback to user behaviour in upday. We surveyed users to ask <em>“what’s the main reason you use upday?”</em>, and they overwhelmingly told us that they use it to quickly check the <strong>current headlines</strong> in the news around them. To stay informed. Not to <strong>relax</strong> with longer pieces or to explore the content recommended by their <strong>social</strong> circles, but to keep abreast of rapidly changing current events. The data seemed to support that – the vast majority of sessions came from users who visited 5-10 times a day, for less than 30 seconds each time. They scrolled the headlines and closed the app again. Case closed.</p>
<p>But where did the other sessions come from? Interesting, the majority of <strong>minutes</strong> spent in the app came from another kind of user – one who enjoyed long, lazy sessions, swiping through hundreds of cards, often later in the evening, 8-11pm. They didn’t seem interested in the headlines, only in consuming many different kinds of content in a relaxed way. Why were their voices not represented in the survey we launched?</p>
<p>By mapping the users to their sessions and plotting them out over the course of a day, it became clear that these slower users were the <strong>same people</strong> as the rapid users who voiced their feedback. During the morning and the work hours, people check the app for headlines frequently and quickly – and when questioned about it, that’s what <strong>they told us</strong> they do. However later in the evening, perhaps on the sofa distractedly watching something on Netflix, they browsed more slowly and less intensively through the app, consuming content in a less engaged way. It didn’t occur to them to tell us about this use case, because that’s <strong>not how they think of themselves</strong> using upday. But this second use case is just as important, and we never would have discovered had we not explored the data describing it. There was a chasm between users’ behaviour and the way users perceived their own behaviour. Discovering this enabled a holistic, <strong>unfiltered</strong> picture of how and why our users use upday.</p>
<h2 id="data-and-truth">Data and truth</h2>
<p>This discrepancy between words and actions is similar to the concept of ‘Reality Mining’ coined by social physicist Alex Pentland. He describes it as <em>“the process of analyzing the patterns within digital breadcrumbs”</em> – or in other words, <strong>looking for meaning in data</strong>. The data we create represent reality, <em>“by recording what each of us has chosen to do. And this is very different from what is put on Facebook; postings on Facebook are what users choose to tell each other, edited according to the standards of the day”</em> (Pentland, 2015).</p>
<p>This concept can be found across domains, including in the discipline of philosophy called <strong>phenomenology</strong>, which studies how humans experience the world and their own being. It’s particularly concerned with the difference between what is <strong>objectively true</strong> and what is <strong>subjectively true</strong>, and how humans experience the objective world through their subjective senses (Owen, 1994). The philosopher Husserl named <strong>noesis</strong> as the <em>“process of consciousness”</em> – i.e. human experience of something, and <strong>noema</strong> as <em>“the object as it is intended”</em> – i.e. the actual reality of the object (Smith, 2016). This could be why our actions don’t match our words, and why we often don’t even realise it; the actual actions we take (noema) and our own perception of those actions (noesis) are just fundamentally different things, and humans aren’t great at bridging the gap. In this case it’s no surprise that our factual data doesn’t always mirror our described perception.</p>
<p>Blending phenomenology and reality mining might enable some new insight. Both are concerned with the difference between <strong>perception and reality</strong>, one from a theoretical point of view and one from a practical point of view. If how we perceive our actions is how we describe them in traditional feedback methodologies, then how they actually are, is reflected in the data we non-deliberately generate, and therefore mining the data for reality would give us the best insight – the other side of the user experience story.
If you want to know how your users are behaving and how they interact with your platform, you have to be able to <strong>listen</strong> to not only what they say but also what they do.</p>
<h2 id="product-management-now">Product Management Now</h2>
<p>In the era of <strong>big data</strong>, better insights are possible. As a PM, you lead the charge of seeking, interpreting and applying user feedback in order to maximize user engagement. This in turn maximizes the business’ potential for success. Product Management Then only required you to be literate in the basic analytical skills required to pursue this goal: identifying the KPIs to improve and repeating the cycle of iterate > seek feedback > assess > adapt. Feedback methodologies like interviews, NPS (Net Promoter Score) surveys and focus groups were staples of the ‘seek feedback’ phase, resulting in insights that were <strong>anecdotal</strong> in nature, <strong>limited</strong> in scope and often <strong>scientifically invalid</strong>, due to being filtered and phrased people with situational motivations.</p>
<p>Product Management Now demands a greater level of data <strong>literacy</strong>, and it demands <strong>curiosity</strong>. Qualitative feedback alone is not enough, the quantitative feedback (data) that users give us must also be explored. This means that you need to be able to ask questions of the data and know how to go about answering them. I propose an extension of the diagram, to look like this:</p>
<center>
<picture>
<a href="/images/blog/product_management_user_feedback/fig2.png"><img src="/images/blog/product_management_user_feedback/fig2.png" alt="Product Management now" /></a>
<figcaption>Fig 2: Product Management Now. A facelift, by me.</figcaption>
</picture>
</center>
<p>User insight encompasses everything from UI design to reality mining. Gathering user feedback now involves not only personal, traditional methods (considered feedback) but <strong>analysis</strong> of data (unconsidered feedback) too.
Strong PMs today rely on measurement, testing and incremental iteration instead of gut feeling or long discussions around a table of managers who are not users. They promote cultures of curiosity and <strong>hypothesis</strong> – and then embark on the exploration needed to prove or disprove.</p>
<p>This doesn’t mean that a PM has to be a fully functional data scientist. Like any of the areas of expertise illustrated in Fig 1, the PM is not a replacement for domain expertise. <strong>Think competency, not fluency</strong>. But the overlap between product and data science is potentially larger than that with business or tech – because many companies do not have the resources to employ full-time data scientists (or have not culturally adapted to the necessity of doing so) the <strong>PO may represent the only data science literacy in the team</strong>. The market seems to recognize this particular compatibility: hired.com 2017 <em>‘State of Global Tech Salaries’</em> report (Kirkpatrick, 2017) noticeably splits the industry into two sections – software engineers, and product managers and data scientists. It also provides an illuminating summary of the most sought-after skill sets from both groups:</p>
<center>
<picture>
<a href="/images/blog/product_management_user_feedback/fig3.png"><img src="/images/blog/product_management_user_feedback/fig3.png" alt="Analysis of desired skill sets in product and data" /></a>
<figcaption>Fig 3: Data scientist Dr. Jessica Kirkpatrick for hired.com’s analysis of desired skill sets in product & data.</figcaption>
</picture>
</center>
<p>For Product Managers, data management is already a demanded skill set. This will only increase – I would not be surprised to see ‘Data Science’ skills like data analysis, SQL and statistics appearing in the ‘Product Management’ list in next year’s report. Let’s examine at a couple of typical PM responsibilities and see how <strong>traditional feedback skills augmented with real data literacy</strong> can enable better results:</p>
<h3 id="audience-development">Audience Development:</h3>
<p>SQL is a database language; it allows you to interact with a database by asking questions of it and receiving results. SQL is one of the best semi-technical ways to <strong>interact with data</strong>, and as a PM the more insight you can extract without dependency on developers, the better. This was perhaps my key learning of 2016, and to act on it I undertook a data management course. Using these seedling skills, I was able to analyse upday’s users in terms of behavior, and start to classify them into ‘<strong>audiences</strong>’, essentially groups of people who share meaningful traits or interests. It’s not an understatement to say that this analysis formed the backbone of our product strategy for advertising. We were able to run campaigns targeted to the audiences I found, which resulted in <strong>engagement rates 3-4 times higher</strong> than untargeted campaigns, meanwhile reducing waste of the advertiser’s dollars and minimizing the users who saw the advertisements – a win-win.</p>
<h3 id="consensus-building">Consensus building:</h3>
<p>Stakeholder management is an everyday part of Product Management. PMs are the <strong>representative of the user</strong> within the company, which is full of people who have different domains of expertise, motivations and opinions. Making sure that the best features get developed requires diplomatic exercise and excellent communication (soft) skills. Often, PMs use visual aids like presentations or diagrams to share their vision more effectively.
But even the most appealing diagram is worthless without the substance of <strong>proof</strong> behind it. Data scientists are experts at data visualization. This is the process of taking what you found in the data and communicating it – it’s like storytelling. DJ Patil, Data Scientist in Residence at Greylock Partners says <em>“a data scientist is that unique blend of skills that can both unlock the insights of data and tell a fantastic story via the data”</em> (Rogers, 2012). Murtaza Haider calls a data scientist <em>“someone who finds solutions to problems by analyzing big or small data … and then tells stories to communicate her findings to the relevant stakeholders”</em> (Haider, 2015). By these definitions, <strong>every PM should be part data scientist too</strong>.</p>
<p>The list can go on: A/B testing is more effective when you have a solid understanding of <strong>statistical relevance</strong> and how to formulate multivariate tests. Personalising user experience to encourage a particular conversion is more effective when your product can dynamically <strong>learn and adapt from user behavior</strong>. Even Agile methodologies are more effective when you can analyse and visualize the productivity data of your own team. In short, a data-literate PM is <strong>more effective</strong> than a qualitative-only PM.</p>
<p>It’s more important than ever for Product Managers to develop the data literacy needed to drive the process of gathering insights: after all, you are the voice of the user, there to understand and represent their needs and wants (Bharathi, 2014). <strong>Qualitative feedback alone is the tip of the iceberg</strong>, and data is emerging as the new channel of user feedback, underneath the explicit feedback we’re used to – and so too must Product Managerss emerge who are equipped to hear it all. It may have taken a couple of decades, but I figured out that the smartest people never stop checking their math.</p>
<h2 id="references">References</h2>
<p>Bharathi, S. (2014, July 9). Who Is Your Product Owner? Retrieved from <a href="https://www.scrumalliance.org/community/articles/2014/july/who-is-your-product-owner" target="blank">Scrum Alliance</a>
<br />
Cheng, G. a. (2016, 06 23). Soft Skills for a Tough World. Retrieved from <a href="https://www.fas.harvard.edu/~sica/2016materials/45_soft_skills.pdf" target="blank">Harvard Faculty of Arts & Sciences</a><br />
Dobelli, R. (2013). The Art of Thinking Clearly. New York: HarperCollins.<br />
Haider, M. (2015). What Makes Someone a Data Scientist? In M. Haider, Getting Started with Data Science (pp. 12-15). IBM Press. Retrieved from <a href="bigdatauniversity.com" target="blank">bigdatauniversity.com</a><br />
Kirkpatrick, J. (2017). 2017 State of Global Tech Salaries. Retrieved from <a href="https://hired.com/state-of-salaries-2017" target="blank">hired.com</a><br />
Lindstrom, L. (n.d.). 7 Skills You Need to Be a Great Product Owner. Retrieved from <a href="https://www.scrumalliance.org/agile-resources/7-skills-you-need-to-be-a-great-product-owner" target="blank">Scrum Alliance</a><br />
Owen, I. R. (1994). Phenomenology - What is it? And what does it do? Retrieved from <a href="http://www.intentionalitymodel.info/pdf/PHWIIWCD.pdf" target="blank">The Intentionality Model</a><br />
Pentland, A. (2015). Social Physics: How Social Networks Can Make Us Smarter. Boston: Penguin Books.<br />
Rogers, S. (2012). What is a Data Scientist? Retrieved from <a href="https://www.theguardian.com/news/datablog/2012/mar/02/data-scientist">theguardian.com</a><br />
Rudder, C. (2015). Dataclysm: Love, Sex, Race, and Identity–What Our Online Lives Tell Us about Our Offline Selves. New York City: Broadway Books.<br />
Smith, D. W. (2016). Phenomenology. Retrieved from <a href="https://plato.stanford.edu/entries/phenomenology/#PhenOntoEpisLogiEthi" target="blank">The Stanford Encyclopedia of Philosophy (Winter 2016 Edition)</a></p>
<div class="footnotes">
<ol>
<li id="fn:1">
<p>Diagram based on Martin Eriksson’s (2011) original, with my commentary added. <a href="#fnref:1" class="reversefootnote">↩</a></p>
</li>
<li id="fn:2">
<p>Named after Matthew McConnaughey’s character from Dazed and Confused <a href="#fnref:2" class="reversefootnote">↩</a></p>
</li>
</ol>
</div>
<p><a href="https://upday.github.io/blog/product_management_user_feedback/">Product Management Then & Now</br>(or, why user feedback sucks)</a> was originally published by upday dev at <a href="https://upday.github.io">upday tech blog - now at medium</a> on March 08, 2017.</p>
https://upday.github.io/blog/team_cut_story2017-03-03T04:00:00+00:002017-03-03T04:00:00+00:00Johannes Theilmannhttps://upday.github.iojohannes.theilmann@upday.com
<p>At upday we challenge ourselves to achieve technical excellence and a strong understanding of business needs. Due to external and internal circumstances we have quite a volatile team setup. At least each quarter we change either the team members, the PO or the whole structure of our teams - from horizontal to cross-functional to fluent and back again. However our goal is to have stable teams with a clear business orientation while also enabling team members to grow technically and to support each other (eg. reviews, testing, tech discussions).</p>
<p>It is the challenge for each team member to adapt quickly to new teams, different skill sets and goals. I would like to share how we approach building stronger teams and to have a checklist for myself when a new team is set up:</p>
<p><strong>As a</strong> Product Owner <strong>I want</strong> to set up teams quickly and efficiently <strong>in order to</strong> get them up ‘n’ running fast and deliver the highest business value.</p>
<p>Acceptance Criteria:</p>
<ul>
<li>Project goals are set for the next quarter</li>
<li>Team set up is agreed with agile coach / scrum master / team-cut captain</li>
<li><a href="http://theteamcanvas.com/use/" target="blank">Team alignment canvas</a> session defines:
<ul>
<li>communication tools (Slack, Skype, email etc.)</li>
<li>planning and progress meetings (Jira, Trello)</li>
<li>roles, component owner, facilitator, lead</li>
<li>team agreement</li>
</ul>
</li>
<li>Depict the process the team is improving on a <a href="http://www.storiesonboard.com" target="blank">story map</a>
<ul>
<li>map projects</li>
</ul>
</li>
<li>Calculate the <a href="http://blackswanfarming.com/cost-of-delay-divided-by-duration/" target="blank">CD3</a> (cost of delay divided by duration) and <a href="http://www.scaledagileframework.com/wsjf/" target="blank">WSJF</a> (weighted short jobs first) with the team.</li>
<li>PO-team choose projects based on CD3 and/or WSJF</li>
<li>Feedback from stakeholders is collected on a confluence page</li>
<li>Epics for projects define clearly user/business value and metrics, objectives and key results (<a href="https://rework.withgoogle.com/guides/set-goals-with-okrs/steps/introduction/" target="blank">OKRs</a>) to measure the success of an epic.</li>
<li>External team members are invited to groomings to make upcoming changes transparent early on and to final reviews to identify potential follow up stories</li>
</ul>
<p>We already made several experiments with the team setup without really being able to measure the outcomes. We also realised that not all our needs are covered by the existing experiments. We decided to measure the success of teams <a href="https://youtu.be/Avs70dZ3Vlk?t=4m27s" target="blank">via following KPIs</a> (introduced by Martin Fowler DevOps state of report 2014):</p>
<ul>
<li><a href="http://kanbantool.com/kanban-library/analytics-and-metrics/kanban-definition-of-lead-time-and-cycle-time#.WHdWULbhDBJ" target="blank">Cycle time</a>
<ul>
<li>How often do we deploy to production?</li>
<li>How much time does it take for a feature from done to production?</li>
</ul>
</li>
<li><a href="http://www.slideshare.net/management30/the-happiness-of-workers/7-Engagement_probablycorrelates_with_satisfactionand_happinessBut" target="blank">Team happiness</a></li>
</ul>
<p>It’s also interesting to visualise the amount of meetings we have, to see how effectively we communicate and how much ‘focus time’ is left for each sprint.</p>
<p>In the next article about team cut we will follow up on our latest <a href="http://upday.github.io/blog/lean_change_insights_board/" target="blank">experiment</a> (running for 3 months) - particularly on the initialisation of the team work and how exactly we measure our experiment KPIs.</p>
<p>What are your experiences in terms of team cuts and measuring team performance?</p>
<p><a href="https://upday.github.io/blog/team_cut_story/">A Team Cut Story</a> was originally published by upday dev at <a href="https://upday.github.io">upday tech blog - now at medium</a> on March 03, 2017.</p>
https://upday.github.io/jobs/be2017-02-15T16:39:55+00:002017-02-15T16:39:55+00:00upday devhttps://upday.github.iodev@upday.com
<h2 id="about-the-role">About the Role</h2>
<p>You are a Software Engineer by heart. You have clean code for breakfast, you domain drive to work, then you continuously integrate and release by lunchtime. In the afternoon you get up-to-date with the latest technologies and before leaving, you improve the monitoring, add a circuit-breaker or do whatever is necessary to have a monkey-proof system you are proud of. You go to bed with the DoD under your pillow and dream of load tests with a smile on your face.</p>
<p><strong>Responsibilities:</strong></p>
<ul>
<li>You strive to create both top-quality code and a great product</li>
<li>You see problems yourself and take ownership in solving them</li>
<li>You and your team are responsible for the entire software development lifecycle from planning to coding to maintenance</li>
</ul>
<p><strong>Qualifications and Skillset:</strong></p>
<ul>
<li>An academic degree in Computer Science, Mathematics, Engineering or equivalent practical experience</li>
<li>Java, Spring, Spring-Boot</li>
<li>Persistence Technologies (SQL, NoSQL)</li>
<li>Test Driven Development (TDD)</li>
<li>Monitoring and maintenance of systems</li>
</ul>
<p><strong>Experience in any of the below would be a plus:</strong></p>
<ul>
<li>Microservices, Spring-Cloud</li>
<li>Continuous Integration/Continuous Delivery</li>
<li>AWS</li>
</ul>
<p><strong>Your Profile:</strong></p>
<ul>
<li>You are looking for a full-time, on-site developer position in the heart of Berlin</li>
<li>You are a team player with experience working in an agile environment</li>
<li>You are keen to develop your skills, and those of your team</li>
<li>You have an interesting hobby such as brewing beer or space travel</li>
</ul>
<p>We expect that development is not new to you and will ask you to show us your skills by taking part in a test to demonstrate your expertise.</p>
<p>You may be young or you may be a veteran in the software development world. This is not important to us.</p>
<p>If this is you, <a href="https://global3.recruitmentplatform.com/syndicated/private/syd_apply.cfm?ID=PPKFK026203F3VBQB8N68V7HK&nPostingTargetID=7238">get in touch with us</a>!</p>
<p><a href="https://upday.github.io/jobs/be/">Software Engineer - Backend</a> was originally published by upday dev at <a href="https://upday.github.io">upday tech blog - now at medium</a> on February 15, 2017.</p>
https://upday.github.io/unpublished/be-s2017-02-15T16:00:00+00:002017-02-15T16:00:00+00:00upday devhttps://upday.github.iodev@upday.com
<h2 id="about-the-role">About the Role</h2>
<p>You are a Software Engineer by heart. You are experienced in backend and mobile development. Others would call you a full-stack developer. You can distinguish between good and bad software architecture. You can even smell it. You have an architectural vision, you have your point, can convince others but not being too bossy at the same time. It’s easy for you to explain technical stuff to techies as well as to non-techies. You know how to code and you do it often!</p>
<p>We have good engineers, but we need someone who adds the little extra on top.</p>
<p><strong>Responsibilities:</strong></p>
<ul>
<li>You keep an overview of backend and mobile architecture</li>
<li>You support and enforce communication between teams regarding architectural decisions and changes</li>
<li>You are an advocate for quality</li>
<li>You ask the right questions (often) and always scrutinize the status quo</li>
<li>You and your team are responsible for the entire software development lifecycle from planning to coding to maintenance</li>
<li>You code… a lot!</li>
</ul>
<p><strong>Qualifications and Skillset:</strong></p>
<ul>
<li>An academic degree in Computer Science, Mathematics, Engineering or equivalent practical experience</li>
<li>Years and years of experience as a developer or whatever qualifies you for the position in question</li>
<li>Understand how to build, release and run a complex software system</li>
<li>Understand CD/CI and build pipelines</li>
<li>Understand test automation</li>
<li>Understand how to scale applications and services</li>
<li>Persistence Technologies (SQL, NoSQL)</li>
</ul>
<p><strong>Experience in any of the below would be a plus:</strong></p>
<ul>
<li>Spring-Cloud</li>
<li>AWS</li>
<li>Android Core, RxJava and Dependency Injection</li>
</ul>
<p><strong>Your Profile:</strong></p>
<ul>
<li>You are looking for a full-time, on-site developer position in the heart of Berlin</li>
<li>You are a team player with experience working in an agile environment</li>
<li>You are keen to develop your skills, and those of your team</li>
<li>You have an interesting hobby such as brewing beer or space travel</li>
</ul>
<p>You may be young or you may be a veteran in the software development world. This is not important to us, but remember this is a senior position and the bar is set high.</p>
<p>If this is you, <a href="mailto:jobs@updudes.net">get in touch with us and send us an email to jobs@updudes.net</a>!</p>
<p><a href="https://upday.github.io/unpublished/be-s/">Senior Software Engineer - Backend and Mobile</a> was originally published by upday dev at <a href="https://upday.github.io">upday tech blog - now at medium</a> on February 15, 2017.</p>
Start: Experiments]]>https://upday.github.io/blog/lean_change_insights_board2017-01-12T04:00:00+00:002017-01-12T04:00:00+00:00Mari Kumlienhttps://upday.github.iomari@upday.com
<p>A new year comes with the feeling that now is the time to change things. How many of us started the year with a promise to change and a few days later, it was already broken? <em>Once a year, big-bang changes don’t work - it never did and never will.</em> Instead, change things when and where you feel the need to. At upday, we strive to continuously improve; how we work and what we deliver. This requires constant change.</p>
<p>How do we foster an organization where the changes are based on what is truly going on in the organization and its surroundings? Where change is not driven by a few, but by anyone who sees the need for change? Where change is not planned upfront and executed top-down but as small steps involving people who are affected? Where we all take part in shaping the future of the organization we want to work in and where we can deliver excellence?</p>
<h2 id="where-did-we-start">Where Did We Start?</h2>
<p>I (Agile Coach at upday) joined a Lean Change Agent workshop together with Richard, one of our Data Scientists, and our CTO Andi. It was a two day workshop held by Mike Weber and based on the book by Jason Little: <a href="http://leanchange.org/lean-change-management/" target="blank"><strong>Lean Change Management</strong></a>. It was an active workshop including lots of participation and interactions, with great tools and ideas about how to work with change in an organization.</p>
<p>We were introduced to a way of approaching change by <strong>collecting insights</strong> and from these, create small <strong>experiments</strong> that you <strong>evaluate and learn</strong> from before you take the next step. The idea is that instead of planning the change upfront, to create small experiments based on your current understanding and insights gathered, and review these after a fixed time period. The word <em>experiment</em> is important here since it emphasizes that until we have tried something out; verified our assumptions (hypotheses) based on predefined metrics, we don’t know whether it’s the right thing to do and if our change will be successful. Another key ingredient is to involve people affected by the change in designing the change. This is called the <a href="http://leanchange.org/resources/lcm/" target="blank">Lean Change Management cycle</a> and it’s a feedback-driven model for managing change.</p>
<center>
<picture>
<a href="/images/blog/lean_change_insights_board/LCM_cycle.png" target="blank"><img src="/images/blog/lean_change_insights_board/LCM_cycle.png" alt="Lean Change Management cycle" /></a>
<a href="http://leanchange.org/resources/lcm/" target="blank"><figcaption>Lean Change Management cycle</figcaption></a>
</picture>
</center>
<h2 id="the-insights-board-was-born">The Insights Board Was Born</h2>
<p>Collecting insights is about trying to understand the current state (of the organization). This does not happen in a meeting room where managers discuss their opinions and ideas based on their assumptions on how the organization is running. To really try to understand, I have to gather opinions, thoughts, reactions from people working <strong>IN</strong> the organization. One way of collecting insights is through an “insights board”.</p>
<p>Inspired by the stories shared to us during the workshop, as we came back to the office, Andi set up our first insights board. He basically did a braindump of his own insights, grouped them and invited others to add, group and vote. It was done very informally and spontaneously by different updudes (as we call ourselves working at upday) or alone if we had the time and inspiration.</p>
<center>
<picture>
<a href="/images/blog/lean_change_insights_board/updudes.jpg" target="blank"><img src="/images/blog/lean_change_insights_board/updudes.jpg" alt="updudes in action at the insights corner" width="250" /></a>
<figcaption>updudes in action at the insights corner</figcaption>
</picture>
</center>
<p>Although we were quite overwhelmed by all the post-its, we felt that what we found there was not news to us. We more or less had a similar perception of how things were working in our organization. We were aware of what our problems were. The question was what to do with this knowledge. It’s quite amazing, we actually had a lot of collective experience, knowledge and feelings about our organization there on the wall and we could just use it to start changing things.</p>
<p>Unfortunately, it wasn’t that simple. We were often too busy with other things that we didn’t take enough time to spontaneously reflect, especially not in a small group. However, we strongly believe that working with insights and experiments is the right way to go and we need to create time for it and continuously improve how we approach change.</p>
<h2 id="iteration-1-introduce-structure">Iteration 1: Introduce Structure</h2>
<p>We started to get a bit frustrated about all the things we wanted to do, but just didn’t manage to. At about the same time, we also realized that we needed to introduce more <strong>structure to the board and the process</strong>; how do we follow-up, what do we do with the insights and outcome of experiments, what about responsibilities? At this point, <a href="https://www.linkedin.com/in/jessicacriddle" target="blank">Jessica Criddle</a> joined us as Agile Coach. She came with a fresh mind, new perspective and lots of energy. This helped a lot to light the spark again. We did a first draft of how the process could look.</p>
<center>
<picture>
<a href="/images/blog/lean_change_insights_board/Process.jpg" target="blank"><img src="/images/blog/lean_change_insights_board/Process.jpg" alt="A first draft of the insights process" width="350" /></a>
<figcaption>A first draft of the insights process</figcaption>
</picture>
</center>
<h2 id="iteration-2-simplify">Iteration 2: Simplify</h2>
<p>The process looked quite good on paper and it did help us to get a clear, shared picture, but it was a bit too much. To actually make the Insights board something we all used, we had to <strong>simplify</strong>. So we did. We cleaned up the board, made some simple rules, made an effort to make it a regular habit to work in front of the board. It was a new start! Two updudes also assigned themselves new roles; “Lord of the board” and “Master of cluster” with the responsibilities to make people gather around the board and help them use it.</p>
<p>When creating an experiment, it is important to include a hypothesis; what outcome do you expect after running the experiment?</p>
<p>“<strong>IF</strong> <em>this is changed</em> <strong>THEN</strong> <em>it will lead to these characteristics.</em>”</p>
<p>You also have to decide how to measure the outcome; when the experiment is over and you review it, how do you know if it succeeded or not? As a means to simplify, we introduced a template for creating experiments, which made it easier to get started. The template included all steps needed to create an experiment and how to review it.</p>
<center>
<picture>
<a href="/images/blog/lean_change_insights_board/Experiment_Template.jpg" target="blank"><img src="/images/blog/lean_change_insights_board/Experiment_Template.jpg" alt="Experiment template" width="350" /></a>
<figcaption>Experiment template</figcaption>
</picture>
</center>
<p>To make it more clear what an experiment could look like, here are a few examples of hypotheses from our experiments:</p>
<ul>
<li>“IF <em>we change the team setup in the suggested way,</em> THEN <em>we gain productivity for delivering on focus areas and our happiness at work will increase.</em>”</li>
<li>“IF <em>team uses End-of-Day emails as daily standup,</em> THEN <em>it will fulfill the purpose of the standup and keep team concentration.</em>”</li>
</ul>
<h2 id="iteration-3-involve-and-empower">Iteration 3: Involve And Empower</h2>
<p>Gradually it spread how to use the insights board and how all updudes could get <strong>involved</strong> in changing upday. We made a collective effort to make the board alive and active and actually use the insights posted there.</p>
<p>It was a positive peer pressure; “join me, let’s spend some minutes at the insights board” after the weekly all-hands or by encouraging each other to use the insights board and create an experiment when something not working so well in the organization was addressed. The idea is to <strong>empower</strong> everyone to do something - <strong>to own the possibility to change and shape our future!</strong></p>
<p>It’s important to note that an experiment could be around anything you believe should change in your organization; culture, processes, tools, roles. It could be something concerning the whole organization or just a few people. The key (again) is not to have people with fancy titles plan upfront at the computer, but to involve people affected by the change in designing it. Empower and encourage them to drive the change!</p>
<h2 id="learnings">Learnings</h2>
<p>Along this journey, we learned a lot about ourselves and how to approach change with the help from an insights board. Using insights works best if we have a high level of trust; we have to be really open and honest with each other when we post insights. This is something we always have to work on and we’ll admit: we are not there yet. Also, to make sense for anyone to post insights, we have to actively use the things posted there. Turn insights into experiments, continuously drive change in small steps, involving people affected. If we don’t, we lose trust in the board and the process and it goes stale.</p>
<center>
<picture>
<a href="/images/blog/lean_change_insights_board/board.jpg" target="blank"><img src="/images/blog/lean_change_insights_board/board.jpg" alt="Current state of our insights board" width="350" /></a>
<figcaption>Current state of our insights board</figcaption>
</picture>
</center>
<p>Another thing that we learned is that it’s easy to lose focus after the experiment is done, but in order to see what works for you and for change to really happen, it’s important to focus on the review. Look at the metrics, evaluate the outcome, and decide on the next steps. <strong>Persevere or pivot</strong>. This, and taking small steps, is two key ingredients to lean change management.</p>
<h2 id="the-future">The Future</h2>
<p>We have approached our insights board in an iterative way. It is an experiment that will gradually evolve. How we use it and how it is designed will change step-by-step as we see what works for us and what doesn’t.</p>
<p>In a future post, we will give more details on where this journey took us: how the insights board and the experiments helped us to improve.</p>
<p><a href="https://upday.github.io/blog/lean_change_insights_board/">Stop: New Year’s Resolutions</br>Start: Experiments</a> was originally published by upday dev at <a href="https://upday.github.io">upday tech blog - now at medium</a> on January 12, 2017.</p>