Google has released the majority of its Google Cloud-based data warehouse portfolio, including Cloud Dataflow, which is designed for third-party integrations and offers more control over cloud environments. Google engineers argue for a shift in data processing approaches, emphasizing the need to handle unbounded datasets with flexible, automated workflows. The new system aims to balance correctness, latency, and cost, providing stability and efficiency for large-scale data deployments. The release includes SDKs for building parallelized data processing pipelines and a managed service for optimizing workflows.

AWS is expected to embrace Docker at its re:Invent event, contrasting with Google's focus on its own container engine technology, Kubernetes. Google Cloud Platform's new insights into Google Container Engine (GKE) emphasize portability and modularity, aiming to support hybrid cloud environments. Shippable, a major Docker ecosystem partner, announced significant funding to expand its product. The article highlights the competitive dynamics between Docker and Google, with AWS showing strong support for Docker's platform.

The article discusses strategies for managing database scalability for real-time transactions, emphasizing the importance of handling volume, latency, and fail-over. It highlights the experiences of companies like Forensiq and 3Scale in scaling their database architecture to manage high-speed, high-volume transactions. The piece also explores the use of sharding and alternatives like Twemproxy, and mentions various database services such as Redis Labs, NuoDB, Aerospike, and SlashDB. The article underscores the need for robust database solutions to support the growing demands of cloud computing, mobile apps, and IoT data.

Google Firebase, presented at the Serverlessconf conference, eliminates the middle-tier by allowing clients to communicate directly with managed services, enhancing app development speed. Firebase offers a real-time database, messaging, authentication, storage, and other services, making it a versatile tool for mobile and web applications. The platform integrates with Google Cloud Platform, providing a scalable infrastructure. Joe Emison of BuildFax praised Firebase for its rapid development capabilities, emphasizing the shift towards serverless technologies and the diminishing role of the middle tier in modern app development.

Amazon Web Services (AWS) identifies five key usage patterns for its Lambda serverless service, emphasizing the benefits of serverless architecture such as scalability, cost-efficiency, and built-in fault tolerance. AWS Solution Architect Cassandra Bonner and other experts discuss the growing adoption of serverless technologies, particularly in hybrid environments where new features are integrated with existing systems. The five serverless patterns include event-driven data processing, web applications, mobile and IoT applications, application ecosystems, and event workflows. Despite some challenges, such as the cost of AWS Step Functions, serverless architecture is maturing and becoming a viable option for businesses beyond startups.

Docker has released new versions of Docker Engine to address security vulnerabilities and improve image validation. The updates include Docker Engine 1.3.3 and 1.4, which introduce an experimental overlay filesystem and enhanced security mechanisms. Marianna Tessel, Docker's senior vice president of engineering, emphasizes the importance of security in Docker's development. The article discusses the inherent security limitations of Linux containers and highlights the need for Docker to engage more with the security community. Partnerships with companies like Red Hat and Canonical are noted as crucial in driving security improvements.

The Social Determinants of Health Screening Tool (SDoHST) underwent qualitative validation at Lyell McEwin Hospital in Adelaide, South Australia, involving patient and stakeholder focus groups. The study, funded by SA Health and the University of Adelaide, led to the SDoHST being the first validated tool in Australia for measuring health-related social needs (HRSNs). The validation process resulted in the addition and removal of items, with the final version comprising 12 items. The tool aims to help healthcare systems identify and address unmet social needs through community service referrals.

Government health officials insist there is no need to close schools or workplaces during the upcoming 41C (106F) heatwave, despite major disruptions in transport services and warnings from the Met Office and UK Health Security Agency. The unprecedented heat has led to a national emergency declaration, with advisories for limited travel and health precautions. The Met Office has issued a red warning for extreme heat, and various organizations, including Network Rail and the NHS, are preparing for significant impacts. The heatwave is attributed to climate change and high-pressure systems bringing hot air from the Sahara.

The cannabis industry is transitioning from black market to legal business, facing challenges in adopting 'white collar' business practices and technology. Nick Kovacevich of Kush Bottles emphasizes the importance of data and software in managing the 'seed to sale' cycle. Legalization has brought new complexities, including the need for auditable records and compliance with varying state laws. Software companies like Greenbits, MJ Freeway, Metrc, and BioTrack are providing solutions, but the industry's federal illegality complicates tax and banking issues. Kush Bottles acts as a consultant, helping businesses integrate necessary software. The future holds potential for growth and consolidation as more states legalize cannabis.

Rocana aims to simplify microservices troubleshooting by eliminating the blame game often seen in IT teams. The company has introduced Rocana One, a free version of its Rocana Ops tool, allowing users to analyze up to 1TB of data daily with unlimited retention. Rocana's tools help consolidate monitoring information, providing powerful search, query capabilities, and anomaly detection. The article also mentions other tools like Spiceworks, Wavefront, and Zabbix, which offer various solutions for IT operations and monitoring.

AppFormix, led by CEO Sumeet Singh, has developed a method to address noisy neighbor problems in container and virtual machine environments using Intel's new Resource Director Technology (RDT) available on Xeon E5 v4 chips. This technology provides visibility into cache and memory bandwidth usage, helping to identify resource-hogging containers or VMs. The solution aims to simplify the complexities of scaled-out systems and improve performance in shared application environments, particularly in public and private cloud infrastructures.

A New York-based startup is introducing a community microgrid in Brooklyn, allowing residents to trade locally produced solar energy using blockchain technology. The Brooklyn Microgrid, developed by LO3 Energy and Consensus Systems, uses the TransActive Grid platform to facilitate secure and transparent energy transactions. The initiative aims to enhance local economic benefits, reduce environmental impact, and provide a more resilient energy system. The project, starting in Gowanus and Park Slope, could potentially disrupt traditional energy models and promote renewable energy adoption.

At the Google GCP Next conference in San Francisco, Google showcased Kubernetes 1.2, emphasizing its ability to perform rolling updates and manage persistent storage without downtime. Greg DeMichillie demonstrated the software's capabilities by handling 20,000 requests-per-second during an on-stage demo. Kubernetes 1.2 introduces features like rolling updates, persistent storage, and multi-zone clustering, enhancing seamless deployment and security. Brendan Burns and Tim Hockin highlighted the software's ability to abstract data center configurations and manage storage automatically. Despite its advancements, concerns about security, particularly the handling of sensitive information, were noted by Richard Steenberg from WePay.

Apache Spark 2.0 is set to enhance real-time data analysis with a new feature called structured streaming, allowing interactive queries against live data. This advancement, introduced by Spark founder Matei Zaharia at the Spark Summit East in New York, aims to simplify the processing of streaming data by eliminating the need for a separate database or data warehouse. Structured streaming integrates streaming with batch jobs and interactive querying, and is accessible through a declarative API. It builds on the Spark SQL Engine and Spark Streaming functionality, and benefits from Project Tungsten's performance optimizations. The 2.0 version, expected around April or May, will introduce structured streaming, with more features to be added in future updates. Project manager Reynold Xin will provide further details at the Summit.

Stateless computing, exemplified by AWS Lambda, is poised to transform software development by abstracting infrastructure management and enabling scalable, flexible application deployment. Key benefits include ease of use, scalability, and the ability to break projects into manageable segments. Companies like Microsoft, Iron.io, and StackHut are also advancing in this space, offering similar services. The adoption of stateless computing is expected to grow, particularly among enterprises handling large-scale data processing. However, not all workloads are suitable for this model, and its future success will depend on overcoming current limitations and integrating with existing infrastructures.

Red Hat's OpenShift 3.1, released during the KubeCon conference in San Francisco, introduces significant advancements by integrating .NET Core, JBoss Fuse, Red Hat Gluster, and Red Hat Ceph. This version enhances the platform's capability to support both stateless and stateful applications, enabling persistent storage for Docker containers without additional plug-ins. The update aims to facilitate the coexistence of legacy and cloud-native applications, particularly benefiting industries with numerous legacy systems. The integration with Kubernetes further supports scalable containerized applications, marking a significant step in Red Hat's PaaS evolution.

Pivotal has announced that its Cloud Foundry platform now supports Docker container images, emphasizing that while Docker is popular, it is one of the least sophisticated container management systems. Pivotal's James Watters highlighted the integration of Docker with Pivotal's advanced container management system, aiming at large enterprises. The article also discusses the importance of Spring Cloud Services over Docker for deploying microservices and mentions Pivotal's partnership with CloudBees for continuous integration and delivery tooling. Pivotal positions itself as providing a sophisticated enterprise solution for managing container formats, likening Docker to a 'PDF' and itself to 'Adobe Photoshop' in the container world.

The article discusses the challenges of deploying a new website built with the Angular framework and how using Node.js can alleviate some of these issues. It highlights the benefits of Node.js, including its ability to handle I/O blocking efficiently and its popularity among major companies like PayPal and Netflix. The article also introduces Dynatrace, a monitoring tool that provides an end-to-end view of application performance, particularly useful for microservice architectures. Daniel Khan from Dynatrace will discuss these features at the upcoming Perform user conference in Orlando.

Immutable infrastructure is revolutionizing the way software is shipped, built, and managed by creating unchanging environments that can be redeployed quickly and efficiently. This approach eliminates many traditional issues associated with mutable infrastructure, such as the need for manual updates and troubleshooting. Continuous integration and delivery processes benefit significantly from immutable infrastructure, allowing for faster and more reliable deployments. Companies like Codeship, Snap CI, and CircleCI are leveraging these advantages to enhance their operations. The shift towards immutable infrastructure promises increased reliability, efficiency, and consistency in application deployment.

Mesosphere and Microsoft have partnered to bring Mesos to Windows, enabling organizations to manage mixed Linux and Windows server environments using the same interfaces and deployment technologies. This integration allows for efficient resource allocation and task scheduling across platforms. Mesosphere's DCOS, available since July 2015, offers a comprehensive solution for data center management, adopted by companies like Verizon. The technology supports Docker containers, application scalability, and enterprise-level data center needs, providing a streamlined workflow for developers working across platforms.

Google has officially launched Google Container Engine, a commercial product version of Kubernetes, marking its transition from beta. The service, which is integrated into the Google Developers Console, allows developers to create clusters for container deployment. Starting November 1, pricing will be implemented, with the first five clusters free and subsequent clusters charged at 15 cents per hour. Google Container Engine is designed to simplify container orchestration, building on Docker's container concept but focusing on the orchestration aspect for business. Craig McLuckie, co-creator of Kubernetes, envisions a future where underlying infrastructure like conventional VMs will be obsolete, replaced by dynamic scheduling and cloud-native computing. Google's role as the steward of the Cloud-Native Computing Foundation positions it to potentially lead the orchestration landscape, competing with Amazon AWS and Microsoft Azure.

Catherine Oliver's article on The New Stack discusses the evolving role of containers and Docker in business environments, emphasizing their benefits beyond the developer community. By enabling technical writers, developer evangelists, and marketing teams to utilize containerized environments, businesses can achieve greater agility and reduced time-to-market. The article highlights how companies like Deis, Blockspring, and SimilarWeb are making advanced tech accessible to non-developers, thereby broadening the conversation around container technology and its applications.

Capital One's Hygieia, showcased at the O'Reilly Open Source Conference, represents a significant step in integrating DevOps within large, regulated industries. The unified dashboard enhances visibility and communication across the software delivery pipeline, emphasizing Agile principles and quality integration with tools like JIRA and Git. Despite some skepticism about its alignment with traditional DevOps, Hygieia focuses on reducing the time between customer feedback and updated functionality. Capital One's commitment to open source and continuous improvement positions them alongside industry leaders like Netflix and Etsy in the DevOps space.

Google has announced the creation of the Cloud Native Computing Foundation (CNCF) under the Linux Foundation, aimed at developing reference implementations for container orchestration platforms, including Kubernetes. The CNCF will work alongside the Open Container Initiative (OCI) to manage standards for container images and runtime. This move follows Google's recent sponsorship of the OpenStack Foundation, highlighting its strategic efforts in cloud computing and containerization. Key stakeholders such as Docker, CoreOS, IBM, and others are involved in these initiatives, emphasizing a collaborative approach to advancing container technology.

Deploying and managing a Hadoop stack can be simplified using Apache Ambari and Apache Brooklyn. Apache Ambari facilitates the provisioning, management, and monitoring of Hadoop clusters, while Apache Brooklyn supports application blueprinting and management. By combining these tools, the deployment process, especially to cloud environments like AWS EC2, becomes more straightforward. The article provides a step-by-step guide to setting up a Hadoop cluster, including installation instructions and configuration tips. Future enhancements include extending YAML parameters, adding sensors for data policies, and effectors for cluster resizing. The latest code is available on GitHub, and Cloudsoft, a sponsor of The New Stack, is actively contributing to these developments.

Millions of asteroids pose potential threats to Earth, prompting NASA to enhance detection methods. A contest organized with TopCoder led to a 15% improvement in asteroid detection algorithms. The winning algorithm, utilizing machine learning techniques, processes astronomical images to identify moving objects, distinguishing asteroids from stars. The project underscores the collaborative effort between professional researchers and programmers, leveraging machine learning to advance space exploration and planetary defense.

Redis is gaining traction as a key component in real-time application processing, with companies like Rackspace and Redis Labs leading its adoption. Redis's popularity is rising, as evidenced by its improved ranking on DB-Engines. Businesses are leveraging Redis for its in-memory data processing capabilities, particularly in gaming and e-commerce. The ecosystem around Redis is expanding, with tools like Celery and Resque enhancing its functionality. Complementary technologies such as Elasticsearch and MongoDB are often used alongside Redis to optimize data handling. Redis is also becoming integral to SaaS platforms, offering solutions for scaling challenges in e-commerce and WordPress installations.

The shift to scaled-out computing has changed how servers and containers are perceived and managed. Bernard Golden discusses the 'pets vs. cattle' meme, which differentiates traditional infrastructure from cloud infrastructure, and extends it to containers, likening them to chickens due to their efficiency and rapid deployment. The article predicts a future with a significantly higher number of containers compared to virtual machines, necessitating a shift from manual to automated IT processes. ActiveState's Stackato is highlighted as a solution designed for this future, promoting full automation throughout the application lifecycle.

Kismatic aims to position itself as the leading commercial open source company for the Kubernetes project, similar to Docker's role in containerization. CEO Patrick Reilly outlines the company's strategy to contribute significantly to Kubernetes, including developing a core web UI for easier cluster management. Kismatic's business model focuses on providing service-level agreements and custom application support for enterprises, particularly in the entertainment and finance sectors. The company faces challenges in promoting containerization and competing with Docker's established ecosystem but remains optimistic about Kubernetes' growth and adoption.

Docker has released its orchestration toolkit, which includes Docker Machine, Docker Swarm, and Docker Compose, to demonstrate its commitment to 100% portability and flexibility for developers. The toolkit allows for the orchestration of multi-container, multi-host applications across various data server hosting infrastructures, supporting hybrid cloud architectures. Docker Machine automates host infrastructure provisioning and Docker Engine installation, while Docker Swarm optimizes infrastructure based on application needs. Docker Compose uses a YAML file for application container management. Mesosphere has integrated with Docker Swarm using the Swarm API, offering an alternative optimization product for large-scale container orchestration. Docker emphasizes its open collaboration with ecosystem partners and the freedom for developers to choose how to build, ship, and run their applications.

Microservices, a technology trend for creating modular and independently deployable services, is discussed with a focus on its adoption challenges and benefits. The article outlines four checklists to guide organizations considering microservices: identifying monolithic application issues, essential components for microservices, strategies for starting with microservices, and potential pitfalls to watch out for. Companies like Nike, Netflix, and Twitter are cited as pioneers in adopting microservices. The author, Vivek Juneja, shares insights based on his experience with cloud platforms and microservices since 2008.

Docker alone is insufficient for deploying apps through a PaaS; the real value lies in container orchestration. John Wetherill from ActiveState demonstrates how their Stackato environment simplifies container orchestration, making it easier to manage and scale applications. Stackato, based on Cloud Foundry and Docker, can be deployed on internal infrastructure or public clouds. The article highlights the benefits of using Stackato for automating container orchestration, which would otherwise require manual effort.

Logentries has introduced a new logging container service leveraging Docker's 1.5 release and its new Stats API. This service aims to simplify logging for DevOps teams by integrating logging functionalities into Docker application deployments. Trevor Parsons from Logentries emphasizes the need for better logging solutions in Docker environments, highlighting the increasing use of logs for analytical and business metrics. The article also discusses the challenges and various approaches to logging in Docker, including insights from other industry experts like Christian Beedgen of Sumo Logic.

Docker's rapid growth has led to mixed experiences in production environments. While some companies like Battlefy and Iron.io have successfully integrated Docker for continuous deployment and microservices architecture, others like XMLDirector have found it unreliable and unusable. Analysts like Gartner acknowledge security challenges but are supportive of Docker's direction. The article highlights various use cases, demonstrating both the potential and the challenges of using Docker in production.

The article explores the competitive landscape of database scalability for real-time transactions, focusing on how companies like FoundationDB, Aerospike, and MemSQL use benchmarking to attract customers. FoundationDB has achieved significant transactional processing speeds, while Aerospike emphasizes cost efficiency and speed at scale. MemSQL highlights its ability to handle mixed workloads. The article discusses the challenges of comparing benchmarks due to differing architectures and metrics, and the need for new industry standards to better evaluate database capabilities.

Facebook's storage strategy focuses on securing user content and optimizing data center efficiency. Initially using standard filers, Facebook transitioned to its Haystack storage service in 2010, which significantly improved photo storage efficiency. The company now employs a tiered storage architecture, with Haystack for hot photos, F4 for warm storage, and offline cold storage for older images. Facebook's cold storage facilities prioritize energy efficiency and data durability, utilizing custom-built servers and disaggregated racks to optimize resource use. The company is exploring 'cold flash' and optical disks for future storage needs, emphasizing the importance of tailored storage solutions over standard architectures.

At DockerCon Europe, Andrew Kennedy from Cloudsoft demonstrated Clocker, a tool that simplifies deploying complex applications to Docker using Apache Brooklyn's blueprints. The demo showcased deploying a Riak cluster and a web app cluster managed by Clocker on SoftLayer's infrastructure. Clocker automates Dockerfile creation, manages Docker hosts, and supports auto-scaling. The deployment uses Weave for network management, allowing developers to run complex apps without port forwarding issues. Cloudsoft and Basho are sponsors of The New Stack.

Steve Wilmott, Co-Founder and CEO of 3Scale, demonstrates APIcast, a cloud-based proxy for managing and securing APIs. The tutorial shows how APIcast can be used to gain visibility, analytics, and protection against brute force attacks for APIs. Wilmott configures APIcast with APIs.io, demonstrating key generation and authentication failure upon key change. APIcast offers rate limiting and a developer portal with a Swagger specification for documentation, free for up to 50,000 transactions per day.

3Scale has launched APIcast, a free, cloud-hosted API gateway service designed for small businesses and non-profits to manage their APIs. APIcast offers features such as rate limiting, report generation, and developer portal creation. It is aimed at organizations with less than 50,000 API calls per day, with the potential to scale to paid versions. The service allows businesses to test and manage their APIs without worrying about backend scaling. APIcast is positioned as a market entry point for businesses to explore API-driven opportunities and is part of 3Scale's broader strategy to support smaller organizations as they grow.

Canonical has introduced Snappy Ubuntu, a cloud-based operating system designed for microservices and containerized applications, particularly supporting Docker. It features transactional updates to enhance security and efficiency, especially in the post-Shellshock era. Snappy Ubuntu aims to simplify software packaging and deployment, making it ideal for developers. The OS is compatible with various cloud environments, including Microsoft Azure, Google Compute Engine, and Amazon Web Services. Key figures like Mark Shuttleworth and Dustin Kirkland emphasize its benefits for modern application development.

Docker, Inc. announced a new suite of open platform and orchestration services for application development and deployment at DockerCon Europe. The services include Docker Machine, Swarm, and Compose, designed to manage environments for Docker applications. The company also released an open API for Docker Machine, with Swarm and Compose APIs expected in the first half of 2015. Ecosystem partners are awaiting these APIs to integrate their products with Docker Engine. Red Hat's Lars Herrmann acknowledged the need for container orchestration services but questioned whether Docker should be the provider.

The 2014 Data Science Survey by O'Reilly Media, released at Strata and Hadoop World, highlights the higher wages for data scientists using open source tools, particularly in the Hadoop ecosystem. The survey of 816 professionals across 53 countries found that open source data tools are crucial, with each tool used in the Hadoop ecosystem adding $1,645 to a data scientist's salary. However, it also revealed a gender pay gap, with women in data science earning $13,000 less than their male counterparts, a disparity not explained by tool use or other factors. The survey also noted the value of data scientists skilled in cloud environments and the popularity of tools like Storm and Spark. Upcoming events like Hack.Summit() aim to address gender and diversity inequality in IT.

Docker has experienced significant growth since its launch, particularly in the last six months, with a 1387% increase in container pulls and a 44% rise in contributors. The Docker Governance Advisory Board (DGAB) includes key figures from Docker and representatives from major corporations and users. The board discussed the challenges of maintaining scalability and developer engagement while fostering an ecosystem. Docker's performance metrics show a balanced contribution from community and staff, with a focus on high-quality standards. Moving forward, Docker plans to establish 'Team Meta' to enhance project effectiveness and share a detailed roadmap to support its open culture.

The New Stack has launched its beta version, focusing on developers and others building services on advanced infrastructure. The platform will cover technology, culture, data research, interviews, and hands-on experiences, with a particular focus on lightweight services and systems. The New Stack aims to provide comprehensive coverage of new stack technologies and their implementation across various industries. The launch is credited to the support of sponsors, colleagues, and friends, with a special mention of Heather Fitzsimmons.