Author Archives: admin

GAE : Objectify : Guia de usuario : Introducción

Introduction

Introduction to Objectify

Objectify is a Java data access API specifically designed for the Google App Engine datastore. It’s goals are to:

  • Be easy to learn and understand.
  • Support all native datastore features in an intuitive, human-friendly way.
  • Model sophisticated, strongly-typed, polymorphic data structures.
  • Enable modular, EJB-like transactional logic.
  • Increase performance and decrease cost through smart, transaction-aware caching.
  • Allow major schema migrations “in-place” with zero downtime.
  • Seamlessly coexist with other datastore tools: the Java Low-Level API, JDO/JPA, Twig, Go, Python DB and NDB.

Objectify provides a level of abstraction that is high enough to be convenient, but low enough not to obscure the key/value nature of the datastore. It is intended to be a Goldilocks API – not too low level, not too high level, just right.

Overview

This is a quick tour of what using Objectify looks like, intended to give you a taste of the framework. Full explanations can be found later in the documentation.

Defining Entities

A simple entity:

import com.googlecode.objectify.annotation.Entity;
import com.googlecode.objectify.annotation.Id;
import com.googlecode.objectify.annotation.Index;

@Entity
public class Car {
    @Id Long id;
    @Index String license;
    int color;
}

Somewhat more complicated:

import com.googlecode.objectify.annotation.AlsoLoad;
import com.googlecode.objectify.annotation.Cache;
import com.googlecode.objectify.annotation.Entity;
import com.googlecode.objectify.annotation.Id;
import com.googlecode.objectify.annotation.IgnoreSave;
import com.googlecode.objectify.annotation.Load;
import com.googlecode.objectify.annotation.OnLoad;

@Entity
@Cache
public class Motorcycle {
    @Id String vin;
    @Load List<Person> owners = new ArrayList<Person>();
    @AlsoLoad("colour") int color;
    @IgnoreSave boolean ktm;

    @OnLoad public void onLoad() {
        if (ktm)
            color = ORANGE;
    }
}

Basic Operations

import static com.googlecode.objectify.ObjectifyService.ofy;

Car porsche = new Car("2FAST", RED);
ofy().save().entity(porsche).now();    // async without the now()

assert porsche.id != null;    // id was autogenerated

// Get it back
Ref<Car> ref = ofy().load().key(Key.create(Car.class, porsche.id));  // Ref is async
Car fetched1 = ref.get();    // Materialize the async value

// More likely this is what you will type
Car fetched2 = ofy().load().type(Car.class).id(porsche.id).get();

// Or you can issue a query
Car fetched3 = ofy().load().type(Car.class).filter("license", "2FAST").first().get();

// Change some data and write it
porsche.color = BLUE;
ofy().save().entity(porsche).now();    // async without the now()

// Delete it
ofy().delete().entity(porsche).now();    // async without the now()

Setup

Setting up Objectify in a project

Add objectify.jar to your project

Add objectify-N.N.N.jar to your project’s WEB-INF/lib directory. There are no other jar dependencies.

If you use Maven, see the MavenRepository.

Enable ObjectifyFilter for your requests

Objectify requires a filter to clean up any thread-local transaction contexts and pending asynchronous operations that remain at the end of a request. Add this to your WEB-INF/web.xml:

<filter>
        <filter-name>ObjectifyFilter</filter-name>
        <filter-class>com.googlecode.objectify.ObjectifyFilter</filter-class>
</filter>
<filter-mapping>
        <filter-name>ObjectifyFilter</filter-name>
        <url-pattern>/*</url-pattern>
</filter-mapping>

Guice Alternative

If you use Guice, you don’t use web.xml to define filters. Add this to your servlet module:

filter("/*").through(ObjectifyFilter.class);

…and this to your business module:

bind(ObjectifyFilter.class).in(Singleton.class);

Enable static imports in Eclipse

This step is optional, but will help prevent you from typing ObjectifyService.ofy() over and over again.

Eclipse does not automatically add static imports when you Organize Imports. By default, it won’t even complete static imports when you type ofy[cmd-space]. If you want to save yourself a lot of typing, add a “Favorite” static import for ObjectifyService.ofy().

Visit Window » Preferences » Java » Editor » Content Assist » Favorites and add:

com.googlecode.objectify.ObjectifyService.ofy

Now, when you type ofy[cmd-space], Eclipse will add a static import for you.

More details can be found in this stackoverflow question.

Building Objectify From Source

If you would like to build Objectify from source, see ContributingToObjectify.

Entities

Defining and registering entities

First define your domain model in Java using Objectify’s annotations, then register these classes with the ObjectifyFactory.

Defining Entities

Entities are simple Java POJOs with a handful of special annotations. Objectify has its own annotations and does NOT use JPA or JDO annotations.

Note that throughout this documentation we will leave off getter and setter methods for brevity.

The Basics

Here is a minimal entity class:

import com.googlecode.objectify.annotation.Entity;
import com.googlecode.objectify.annotation.Id;
import com.googlecode.objectify.annotation.Ignore;

@Entity
public class Car {
    @Id Long id;
    String vin;
    int color;
    byte[] rawData;
    @Ignore irrelevant;

    private Car() {}

    public Car(String vin, int color) {
        this.vin = vin;
        this.color = color;
    }
}

Observe:

  • Entity classes must be annotated with @Entity.
  • Objectify persists fields and only fields. It does not arbitrarily map fields to the datastore; if you want to change the way a field is stored, rename the field. Getters and setters are ignored so you can isolate the public interface of your class (eg, public String getVehicleIdentificationNumber() { return vin; }).
  • Objectify will not persist static fields, final fields, or fields annotated with @Ignore. It will persist fields with the transient keyword, which only affects serialization.
  • Entities must have one field annotated with @Id. The actual name of the field is irrelevant and can be renamed at any time, even after data is persisted. This value (along with the kind ‘Car’) becomes part of the Key which identifies an entity.
  • The @Id field can be of type Longlong, or String. If you use Long and save an entity with a null id, a numeric value will be generated for you using the standard GAE allocator for this kind. If you use String or the primitive long type, values will never be autogenerated.
  • You can persist any of the core value types, Collections (ie Lists and Sets) of the core value types, or arrays of the core value types. You can also persist properties of type Key (the native datastore type) and Key<?>) (the Objectify generified version). There are more advanced types that can be persisted which will be discussed later.
  • There must be a no-arg constructor (or no constructors – Java creates a default no-arg constructor). The no-arg constructor can have any protection level (private, public, etc).
  • String fields which store more than 500 characters (the GAE limit) are automatically converted to Text internally. Note that Text fields, likeBlob fields, are never indexed (see Queries).
  • byte[] fields are automatically converted to Blob internally. However, Byte[] is persisted “normally” as an array of (potentially indexed)Byte objects. Note that GAE natively stores all integer values as a 64-bit long.

Polymorphism

Objectify lets you define a polymorphic hierarchy of related entity classes, and then load and query them without knowing the specific subtype. Here are some examples:

import com.googlecode.objectify.annotation.Entity;
import com.googlecode.objectify.annotation.EntitySubclass;
import com.googlecode.objectify.annotation.Id;

@Entity
public class Animal {
    @Id Long id;
    String name;
}

@EntitySubclass(index=true)
public class Mammal extends Animal {
    boolean longHair;
}

@EntitySubclass(index=true)
public class Cat extends Mammal {
    boolean hypoallergenic;
}

Things to note:

  • The root of your polymorphic hierarchy must be annotated with @Entity.
  • All polymorphic subclasses must be annotated with @EntitySubclass.
  • You can skip @EntitySubclass on intermediate classes which will never be materialized or queried for.
  • You should register all classes in the hierarchy separately, but order is not important.
  • Polymorphism applies only to entities, not to @Embed classes.

In a polymorphic hierarchy, you can load() without knowing the actual type:

Animal annie = new Animal();
annie.name = "Annie";
ofy().save().entity(annie).now();

Mammal mam = new Mammal();
mam.name = "Mam";
m.longHair = true;
ofy().save().entity(mam).now();

Cat nyan = new Cat();
nyan.name = "Nyan";
nyan.longHair = true;
nyan.hypoallergenic = true;
ofy().save().entity(nyan).now();

// This will return the Cat
Animal fetched = ofy().load().type(Animal.class).id(nyan.id).get();

// This query will produce three objects, the Animal, Mammal, and Cat
Query<Animal> all = ofy().load().type(Animal.class);

// This query will produce the Mammal and Cat
Query<Mammal> mammals = ofy().load().type(Mammal.class);

Native Representation

When you store a polymorphic entity subclass your entity is stored with two additional, hidden synthetic properties:

  • ^d holds a discriminator value for the concrete class type. This defaults to the class shortname but can be modified with the@EntitySubclass(name="alternate") annotation.
  • ^i holds an indexed list of all the discriminators relavant to a class; for example a Cat would have [[“Mammal”, “Cat]]. Note that subclasses are not indexed by default. See below.

The indexed property is what allows polymorphic queries to work. It also means that you cannot simply change your hierarchy arbitrarily and expect queries to continue to work as expected – you may need to re-save() all affected entities to rewrite the indexed field.

There are two ways you can affect this:

  1. You can control indexing of subclasses by specifying @EntitySubclass(index=true)Subclasses are not indexed by default, you must explicitly enable this for each subclass. Note that this only affects queries; you can always get-by-key or get-by-id and receive the proper typed object irrespective of indexing.
  2. You can use @EntitySubclass(alsoLoad="OldDiscriminator") to “reclaim” old discriminator values when changing class names. Note that this will not help with query indexes, which must be re-saved().

Embedding

Objectify supports embedded classes and collections of embedded classes. This allows you to store structured data within a single POJO entity in a way that remains queryable. With a few limitations, this can be an excellent replacement for storing JSON data.

Embedded Classes

You can nest objects to any arbitrary level.

import com.googlecode.objectify.annotation.Embed;
import com.googlecode.objectify.annotation.Entity;
import com.googlecode.objectify.annotation.Id;

@Embed
class LevelTwo {
    String bar;
}

@Embed
class LevelOne {
    String foo;
    LevelTwo two
}

@Entity
class EntityWithEmbedded {
    @Id Long id;
    LevelOne one;
}

Embedded Collections and Arrays

You can place @Embed objects in collections and arrays:

@Entity
class EntityWithEmbeddedCollection {
    @Id Long id;
    List<LevelOne> ones = new ArrayList<LevelOne>();
}

Some things to keep in mind:

  • This does not support two-dimensional structures of any kind.
    • You cannot nest @Embed arrays/collections inside other @Embed arrays/collections.
    • You cannot put arrays/collections of native types inside @Embed arrays/collections.
  • You can, however, nest @Embed arrays/collections inside any number of @Embed classes.
  • You should initialize collections. Null or empty collections are not written to the datastore and therefore get ignored during load. Furthermore, the concrete instance will be used as-is, allowing you to initialize collections with Comparators or other state.

Native Representation

You may wish to know how @Embed classes are persisted so that you an access them through the Low-Level API. Here is an example, given the example data structures specified above.

EntityWithEmbedded ent = new EntityWithEmbedded();
ent.one = new LevelOne();
ent.one.foo = "Foo Value";
ent.one.two = new LevelTwo();
ent.one.two.bar = "Bar Value";

ofy().save().entity(ent);

This will produce an entity that contains:

one.foo “Foo Value”
one.two.bar “Bar Value”

For collections and arrays of @Embed classes, the storage mechanism is more complicated:

EntityWithEmbeddedCollection ent = new EntityWithEmbeddedCollection();
for (int i=1; i<=4; i++) {
    LevelOne one = new LevelOne();
    one.foo = "foo" + i;
    one.two = new LevelTwo();
    one.two.bar = "bar" + i;

    ent.ones.add(one);
}

ofy().save().entity(ent);

This will produce an entity that contains:

ones.foo ["foo1", "foo2", "foo3", "foo4"]
ones.two.bar ["bar1", "bar2", "bar3", "bar4"]

This is what the entity would look like if the second and third values in the ones collection were null:

ones.foo^null [1, 2]
ones.foo ["foo1", "foo4"]
ones.two.bar ["bar1", "bar4"]

The synthetic ^null property only exists if the collection contains nulls. It is never indexed.

Serializing

An alternative to @Embed is to @Serialize, which will let you store nearly any Java object graph.

@Entity
class EntityWithSerialized {
    @Id Long id;
    @Serialize Map<Object, Object> stuff;
}

There are some limitations:

  • All objects stored in the graph must follow Java serialization rules, including implement java.io.Serializable.
  • The total size of an entity cannot exceed 1 megabyte. If your serialized data exceeds this size, you will get an exception when you try to save() it.
  • You will not be able to use the field or any child fields in queries.
  • As per serialization rules, transient (the java keyword) fields will not be stored.
  • All Objectify annotations will be ignored within your serialized data structure. This means @Ignore fields within your serialized structure will be stored!
  • Java serialization data is opaque to the datastore viewer and other languages (ie GAE/Python). You will only be able to retrieve your data from Java.

However, there are significant benefits to storing data this way:

  • You can store nearly any object graph – nested collections, circular object references, etc. If Java can serialize it, you can store it.
  • Your field need not be statically typed. Declare Object if you want.
  • Collections can be stored in their full state; for example, a SortedSet will remember its Comparator implementation.
  • @Serializ collections can be nested inside @Embed collections.

You are strongly advised to place serialVersionUID on all classes that you intend to store as @Serialize. Without this, any change to your classes will prevent stored objects from being deserialized on fetch. Example:

class SomeStuff implements Serializable {
    /** start with 1 for all classes */
    private static final long serialVersionUID = 1L;

    String foo;
    Object bar;
}

@Entity
class EntityWithSerialized {
    @Id Long id;
    @Serialize SomeStuff stuff;
}

NOTA:

The serialization runtime associates with each serializable class a version number, called a serialVersionUID, which is used during deserialization to verify that the sender and receiver of a serialized object have loaded classes for that object that are compatible with respect to serialization. If the receiver has loaded a class for the object that has a different serialVersionUID than that of the corresponding sender’s class, then deserialization will result in an InvalidClassException. A serializable class can declare its own serialVersionUID explicitly by declaring a field named “serialVersionUID” that must be static, final, and of type long:

ANY-ACCESS-MODIFIER static final long serialVersionUID = 42L;

If a serializable class does not explicitly declare a serialVersionUID, then the serialization runtime will calculate a default serialVersionUID value for that class based on various aspects of the class, as described in the Java(TM) Object Serialization Specification. However, it is strongly recommended that all serializable classes explicitly declare serialVersionUID values, since the default serialVersionUID computation is highly sensitive to class details that may vary depending on compiler implementations, and can thus result in unexpected InvalidClassExceptions during deserialization. Therefore, to guarantee a consistent serialVersionUID value across different java compiler implementations, a serializable class must declare an explicit serialVersionUID value. It is also strongly advised that explicit serialVersionUID declarations use the private modifier where possible, since such declarations apply only to the immediately declaring class–serialVersionUID fields are not useful as inherited members.

 

Compression

You can automatically zip the serialized blob:

@Entity
class EntityWithSerialized {
    @Id Long id;
    @Serialize(zip=true) Map<Object, Object> stuff;
}

You can add or remove zipping at any time. Irrespective of the zip value, Objectify will read the field correctly whether it was zipped or not in the datastore. Of course, when the entity is saved, the zip value will determine whether or not the data in that instance is compressed.

Relationships

A relationship is simply a key stored as a field in an entity. Objectify provides several facilities to make it easier to manage relationships using keys.

Keys

There are two different Key classes in Objectify:

  1. com.google.appengine.api.datastore.Key, usually referred to as Key
  2. com.googlecode.objectify.Key, usually referred to as Key<?>

#1 is the native datastore Key class used by the low-level API. It is the “one true Key” because all datastore operations use this type; Objectify always converts to (or from) the native Key. However, because this structure lacks generic type information, it should almost never appear in your code.

#2 is Objectify’s generified version of Key. It’s much less errorprone to use Key<Person> or Key<Car>, so you should always use Key<?> when you use keys directly.

For all intents and purposes, these structures are equivalent. For example, these two classes are identical in the datastore:

import com.google.appengine.api.datastore.Key;

@Entity
class Thing {
    @Id Long id;
    Key other;
}
import com.googlecode.objectify.Key;

@Entity
class Thing {
    @Id Long id;
    Key<Other> other;
}

You can change the type of the other field at any time, even after data is stored. Remember, the datastore contains the native Key.

Ref<?>s

Even Key<?>s are not very convenient when you are working with graphs of entities. Objectify provides Ref<?>, which works just like a Key<?> but can hold a reference to the actual entity object as well:

import com.googlecode.objectify.Ref;

@Entity
class Car {
    @Id Long id;
    Ref<Person> driver;    // Person is an @Entity
}

Car car = new Car();
car.driver = Ref.create(driverKey);
ofy().save().entity(car).now();

Car fetched = ofy().load().entity(car).get();
ofy().load().ref(fetched.driver);
Person driver = fetched.driver.get();

Ref<?> fields are stored as native Key fields in the datastore. You can freely swap between KeyKey<?>, and Ref<?> in your Java data model without modifying stored data.

This provides a nice consistent structure for your object graph, but it’s still clunky. Ref<?> works best in concert with the @Load annotation.

@Load

import com.googlecode.objectify.annotation.Load;

@Entity
class Car {
    @Id Long id;
    @Load Ref<Person> driver;    // Person is an @Entity
}

Car car = new Car();
car.driver = Ref.create(driverKey);
ofy().save().entity(car).now();

Car fetched = ofy().load().entity(car).get();
Person driver = fetched.driver.get();

Objectify will translate this into an optimal number of batch fetches. For example:

ofy().load().keys(car1, car2, car3);

This will produce one round of batch fetching for the cars, plus one round of batch fetching for all the drivers, plus one round of fetching for any@Load fields in all the drivers… and on. Objectify is also smart about not re-fetching anything that has already been fetched in an earlier cycle. This optimized behavior is very hard to write by hand.

Often you will not wish to @Load every entity relationship every time. @Load provides the ability to specify conditional load groups. See <insert reference to load group section> for more.

Hiding Ref<?>

You will generally find it convenient to encapsulate Ref<?> behind getter and setter methods:

@Entity
class Car {
    @Id Long id;
    @Load Ref<Person> driver;    // Person is an @Entity

    public Person getDriver() { return driver.get(); }
    public void setDriver(Person value) { driver = Ref.create(value); }
}

@Parent Relationships

Parent relationships are described in Concepts. Objectify models them with the @Parent annotation:

import com.googlecode.objectify.annotation.Entity;
import com.googlecode.objectify.annotation.Id;
import com.googlecode.objectify.annotation.Parent;

@Entity
public class Person {
    @Id Long id;
    String name;
}

@Entity
public class Car {
    @Parent Key<Person> owner;
    @Id Long id;
    String color;
}

Each Car entity is part of the parent owner’s entity group. Important things to note:

  • There can only be one @Parent field in a class!
  • Along with the @Id and kind, the @Parent field defines the key (identity) of the entity.
    • Construct a key to the car like this: Key.create(ownerKey, Car.class, carId)
  • The name of the field is irrelevant and can be changed at any time without modifying stored data.
  • The @Parent field can be KeyKey<?>, or Ref<?>.
  • @Load annotations work with Ref<?>s. As a bonus, the parent will be loaded in the same batch as the initial load.

Note that this example is an inappropriate use of the @Parent entity; if a car were to be sold to a new owner, you would need to delete the Car and create a new one. It is often better to use simple nonparent key references even when there is a conceptual parent-child or owner-object relationship; in that case you could simply change the owner.

VERY IMPORTANTIf you load() an entity, change the @Parent field, and save() the entity, you will create a new entity. The old entity (with the old parent) will still exist. You cannot simply change the value of a @Parent field. This is a fundamental aspect of the appengine datastore; @Parent values form part of an entity’s identity.

One-To-Many Relationships

The datastore can natively persist collections of simple types, including keys. This creates one approach for defining one-to-many (and many-to-many) relationships.

@Entity
public class Employee {
    @Id String name;
    List<Key<Employee>> subordinates = new ArrayList<Key<Employee>>();
}

This works with Keys, Key<?>s, and Ref<?>s. You can use @Load annotations with Ref<?>s.

There are some considerations:

  • Every time you load() or save() an Employee, it will fetch and store the entire list of subordinates keys. Large numbers of subordinates will bloat the entity and impact performance.
  • App Engine limits you to 5,000 entries in a multi-valued property.
  • Because fetches are performed by key, you have a strongly-consistent view of the a manager and subordinates.

If you put @Index on subordinates, you can issue a query like this:

Iterable<Employee> managers = ofy().load().type(Employee.class).filter("subordinates", fred);

One-To-Many Alternative

Storing keys in a collection field is a very “appenginey” way of modeling data, and faithfully works with the key/value nature of the datastore. However, this is not the traditional way of modeling one-to-many relationships in a relational database. You can still use the RDBMS approach, but there are caveats:

@Entity
public class Employee {
    @Id String name;
    @Index Key<Employee> manager;
}

Iterable<Employee> subordinates = ofy().load().type(Employee.class).filter("manager", fred);
  • This is a query and therefore weakly consistent.
  • You cannot use this query inside of a transaction.
  • Queries are less cache-friendly than direct key references.

The decision of which approach to take will depend heavily on the shape of your data and what queries you need to perform.

Registering Entities

Before you use Objectify to load or save data, you must register all entity classes for your application. Objectify will introspect these classes and their annotations to build a metamodel which is used to efficiently manipulate entities at runtime.

ObjectifyService.register(Car.class);
ObjectifyService.register(Motorcycle.class);
  • Registration must be done at application startup, before Objectify is used.
  • Registration must be single-threaded. Do not register() from multiple threads.
  • All entity classes must be registered, including polymorphic subclasses.
  • @Embed classes do not need to be registered.

We recommend that you register classes in a static initializer for something that is used at application startup. For example:

public class MyStartupServlet {
    static {
        ObjectifyService.register(Car.class);
        ObjectifyService.register(Motorcycle.class);
    }

    ...the rest of your servlet
}

Because this registration process requires loading and introspecting classes, it adds a small delay to your application start time which is directly proportional to the number of classes you register. While Objectify tries to minimize this delay by eschewing classpath-scanning, the delay is not entirely avoidable (classloading on appengine is very slow). The best that can be said is that this delay is significantly shorter than the equivalent JDO/JPA delay.

GAE : Objectify : Introducción

Objectify is a Java data access API specifically designed for the Google App Engine datastore. It occupies a “middle ground”; easier to use and more transparent than JDO or JPA, but significantly more convenient than the Low-Level API. Objectify is designed to make novices immediately productive yet also expose the full power of the GAE datastore.

  • Objectify lets you persist, retrieve, delete, and query your own typed objects.
  • @Entity
    class Car {
        @Id String vin; // Can be Long, long, or String
        String color;
    }
    
    ofy().save().entity(new Car("123123", "red")).now();
    Car c = ofy().load().type(Car.class).id("123123").get();
    ofy().delete().entity(c);
  • Objectify surfaces all native datastore features, including batch operations, queries, transactions, asynchronous operations, and partial indexes.
  • Objectify provides type-safe key and query classes using Java generics.
  • Objectify provides a human-friendly query interface based on GAE/Python’s Query class.
  • Objectify can automatically cache your data in memcache for improved read performance.
  • Objectify can store polymorphic entities and perform true polymorphic queries.
  • Objectify provides a simple, easy-to-understand transaction model.
  • Objectify provides built-in facilities to help migrate schema changes forward.
  • Objectify has no external jar dependencies beyond the GAE SDK. Just add objectify.jar to your project.
  • Objectify provides thorough documentation of concepts as well as use cases.
  • Objectify has an extensive test suite to prevent regressions.

After you’ve banged your head against JDO and screamed “Why, Google, why??” enough times, start with the Concepts.

You may also wish to read the IBM developerWorks article Twitter Mining with Objectify-Appenginepart 1 and part 2.

Downloads

Version 4 and later of Objectify is released to the Maven Central Repository and can be downloaded directly from there. Older releases can be found in the Google Code Downloads.

 

Concepts

Fundamental Concepts of the Datastore
This is a combined introduction to Objectify and to the App Engine datastore.

So, you want to persist some data. You’ve probably looked at the datastore documentation and thought “crap, that’s complicated!” Entities, query languages, fetch groups, detaching, transactions… hell those things have bloody lifecycles! However, the complexity of JDO is hiding a lot of simplicity under the covers.

The first thing you should do is set aside all your preconceived notions about relational databases. The GAE datastore is not an RDBMS. In fact, it acts much more like a HashMap that gives you the additional ability to index and query for values. When you think of the datastore, imagine a persistent HashMap.

Entities

This document will talk a lot about entities. An entity is an object’s worth of data in the datastore. Using Objectify, an entity will correspond to a single POJO class you define. In the datastore, an entity is a HashMap-like object of type Entity. Conceptually, they are the same.

Since the datastore is conceptually a HashMap of keys to entities, and an entity is conceptually a HashMap of name/value pairs, your mental model of the datastore should be a HashMap of HashMaps!

Operations

There are only four basic operations in the datastore, and any persistence API must boil operations down to:

  • put() an entity whole in the datastore. You can store many at a time.
  • delete() an entity from the datastore, using its identity. You can delete many at a time.
  • get() an entity whole from the datastore, using its identity. You can get many at a time.
  • query() for entities matching indexed criteria you define.

Objectify uses slightly different terminology, but is implemented with the four basic operations above:

  • save() is a put()
  • delete() is a delete()
  • load() may be a get() or a query(), depending on the request

Keys

All entities have either a Long id or a String name, but those values are not unique by themselves. In the datastore, entities are identified by the id (or name) and a kind, which corresponds to the type of object you are storing. So, to get Car #959 from the datastore, you need to call something equivalent to get_from_datastore("Car", 959) (not real code yet).

By the way, I lied. There is actually a third value which is necessary to uniquely identify an entity, called parent. Parent defines a special kind of relationship, placing the child in the same entity group as the parent. Entity groups will be discussed next in the section on Transactions, but for now what you need to know is that this parent (which is often simply null, creating an unparented, root entity) is also required to uniquely identify an entity. So, to get Car #959 from the datastore, you actually need to call something equivalent to get_from_datastore("Car", 959, null)or get_from_datastore("Car", 959, theOwner). Yeech.

Instead of passing three parameters around all the time, the datastore wraps these values into a single object – a Key. That’s all a Key is, a holder for the three parts that uniquely identify an entity.

The native Datastore Key class is simple and untyped, like the native Entity class. Objectify provides a generified Key that carries type information:

Key<Car> rootKey = Key.create(Car.class, 959);
Key<Car> keyWithParent = Key.create(parent, Car.class, 959);

In Objectify, you define your object as a Java class with a mandatory identifier (Long, long, or String) and an optional parent. However, when you look up or reference your object, you do so by Key. In fact, you can (and should) batch together a variety of requests into a single call, even if it will fetch many different kinds of objects:

Map<Key<Object>, Object> lotsOfThings = ofy().load().keys(carKey, airplaneKey, chairKey, personKey, yourMamaKey);

Actually, I lied again. We don’t force you to create keys by hand all the time. There is a convenient shorthand for the very common case of loading a single type of object, but don’t forget that this is really just creating an Key and get()ing it under the covers!

Car c = ofy().load().type(Car.class).id(959).get();
Map<Long, Car> cars = ofy().load().type(Car.class).ids(959, 911, 944, 924);

By the way, Key is used for relationship references as well. Remember that value that defines a parent entity? The type of this parent is Key:

// One of the overloads for static method Key.create():
public static <T> Key<T> create(Key<?> parent, Class<? extends T> kindClass, long id) { ... }

When you create relationships to other entities in your system, the type of the entity relationship should be Key.

Transactions

Transactions are the bread-and-butter of most business processing apps, and most RDBMSes have refined the transaction to an art form. However, GAE’s scalable, distributed datastore presents certain challenges which don’t exist in a single Postgres instance. Transactions on GAE are not quite like RDBMS transactions. Here’s what you need to know:

Entity Groups

When you save() your entity, it gets stored somewhere in a gigantic farm of thousands of machines. In order to perform an atomic transaction, the datastore requires that all the data that is a part of that atomic transaction live on the same server. To give you some control over where your data is stored, the datastore has the concept of an entity group.

Remember the parent that is part of a Key? If an entity has a parent, it belongs to the same entity group as its parent. If an entity does not have a parent, it is the “root” of an entity group, and may be physically located anywhere in the cluster. All its child entities (and children of those children) are part of the same entity group and colocated in the same part of the datastore.

Note that an entity group is not defined by a class, but by an instance! Each instance of an entity class which has a null (or absent) parent defines the root of a separate entity group.

Within a normal transaction, you can only access data from a single entity group. If you try to access data from multiple entity groups, you will get an Exception. This means you must pick your entity groups carefully, usually to correspond to the data associated with a single user. This is dramatically constraining compared to the RDBMSes you are probably used to, where any piece of data is fair game for a single commit.

GAE does offer the ability to perform cross-group (XG) transactions, and Objectify enables this by default. Just as each entity group is analogous to a separate database server, an XG transaction is analogous to a two-phase-commit distributed transaction – and this is exactly how it is implemented. GAE allows you to enlist up to five entity groups in a single transaction, and the more you add the slower it will be. However, this makes creating robust business logic dramatically easier.

Why not store all your data with a common parent, putting it all in a single entity group? You can, but it’s a bad idea. Google limits the number of requests per second that can be served by a particular entity group.

It is worth mentioning that the term parent is somewhat misleading. There is no “cascading delete” in the datastore; if you delete the parent entity it will NOT delete the child entities. For that matter, you can create child entites with a parent Key (or any other key as a member field) that points to a nonexistant entity! Parent is only important in that it defines entity groups; if you do not need transactions across several entities, you may wish to use a normal nonparent key relationship – even if the entities have a conceptual parent-child relationship.

Optimistic Concurrency

When any entity group is changed, a timestamp is written to that group. The timestamp is for the whole group; it is updated when any entity in that group is written.

When you start a transaction, each entity group you touch (by loading, saving, or deleting, up to the XG limit of 5 groups) is enlisted in the transaction. You can make as many changes to entities in those groups as you want. When the transaction is committed, GAE checks the timestamps on all EGs enlisted in the transaction. If any of those timestamps have changed (because some other transaction updated the EG), the whole transaction is rolled back. The commit operation will throw a ConcurrentModificationException.

This is called “optimistic concurrency” or “optimistic locking” because it (optimistically) assumes that transactions will not interfere with each other and simply detects the case, throwing an error. The opposite, “pessimistic locking”, explicitly locks accessed resources and blocks other transactions who try to access the same resources.

Optimistic concurrency tends to have much higher throughput rates and eliminates the deadlocks that plague pessimistic strategies. However, optimistic concurrency means that any transaction may fail with ConcurrentModificationException even though your logic is flawless; the data just happened to be modified while your transaction was in progress.

This means that transactions on GAE need two behaviors:

  • Transactions must be idempotent – you should be able to run the code any number of times and achieve the same result.
  • Transactions must be retried on ConcurrentModificationException – when there is a transaction collision, the right thing to do is just try again. Even under contention, at least one transaction will succeed each round.

Objectify’s Transactions are specifically designed to make this process easy and transparent. You define your transactions as idempotent lambda expressions (or as close to it as Java gets – anonymous inner classes); Objectify will take care of concurrency failures and retries for you.

Transaction Limitations

When you execute a datastore operation, you will either be in a transaction or you will not. If you execute within a transaction:

  • Each EG you touch via get/put/delete/query enlists that EG in the transaction.
    • You can enlist up to 5 EGs, but single-EG transactions are fastest
  • Queries must include an ancestor which defines the EG in which to search. You cannot query across EGs at all, not even in XG transactions.
  • There are some quirks at the low-level API: For example, get()s and query()s will see the datastore “frozen in time” and will not reflect updates even within the transaction. Objectify hides this behavior from you; subsequent fetches will see the same data seen (or updated) previously.

Transactionless

If you operate on the datastore without an explicit transaction, each datastore operation is treated like a separate little transaction which is retried separately. Note that batch operations are not transactional; each entity saved in a batch put() is effectively its own little transaction. If you perform a batch save entities outside of a transaction, it is possible for some to succeed and some to fail. An exception from the datastore does not indicate that all operations failed.

Indexes

When using a traditional RDBMS, you become accustomed to issuing any ad-hoc SQL query you want and letting the query planner figure out how to obtain the result. It may take twelve hours to linear scan five tables in the database and sort the 8 gigabyte result set in RAM, but eventually you get your result! The appengine datastore does NOT work this way.

Appengine only allows you to run efficient queries. The exact meaning of this limitation is somewhat arbitrary and changes as Google rolls out more powerful versions of the query planner, but generally this means:

  • No table scans
  • No joins
  • No in-memory sorts

The datastore query planner really only likes one operation: Find an index and walk it in-order. This means that for any query you perform, the datastore must already contain properly ordered index on the field or fields you want to filter by! And since appengine doesn’t do joins, queries are limited to what you can stuff into a single index — you can’t, for example, filter by X > 5 and then sort by Y.

Actually, it’s not quite true that appengine won’t do joins. It will do one kind of join – a “zig-zag” merge join which lets you perform equality filters on multiple separate properties. But this is still an efficient query – it walks each of the property indexes in order without buffering chunks of data in RAM.

What you should be getting out of this is that if you want queries, you need indexes tailored to the queries you want to run.

To make this easier, the datastore has an innate ability to store each and every (single) property as “indexed” or “unindexed” (Entity.setProperty() vs Entity.setUnindexedProperty(). This allows you to easily issue a queries based on single properties. By default, Objectify does not index properties unless you explicitly flag the field (or class) with the @Index annotation.

To run queries by filtering or sorting against multiple properties (that is, if it can’t be satisfied by a zigzag merge on single-property indexes), you must create a multi-value index in your datastore-indexes.xml. There is a great deal written on this subject; we recommend How Entities and Indexes are Stored and Index Building.

Note that there are some tricks to creating indexes:

  • Single property indexes are created/updated when you save an entity. Let’s say you have a Car with a color property. If you save a Car with color unindexed, that entity instance will not appear in queries by color. To index this entity instance, you must resave the entity.
  • Multi-property indexes are built on-the-fly by appengine. You can add new indexes to your datastore-indexes.xml and appengine will slowly build a brand-new index – possibly taking hours or days depending on total system load (index-building is a low-priority task).
  • In order for an entity to be included in a multi-property index, each of the relevant individual properties must have a single-property index. If your Car has a multi-property index on color and brand, an individual car will not appear in the multi-property index if it is saved with an unindexed color.

 

GAE: Tutorial: Code Lab Exercise 2: Transactions, Memcache and Relationships

Goal

This tutorial will demonstrate the following:

  • Memory Cache using Google Memory Caching Service
  • Handling Transactions
  • Model the relationships across the entities

This code lab is an extension to the previous code lab, adding transactions, caching & relationship management. This exercise builds on the previous product catalog, with entities Customer and Order added to demonstrate a sample store application.

Time for Completion

Approximately 45 minutes.

Prerequisites

Before you begin this tutorial, you should be familiar with:

  • Web application basics and deployment process
  • Servlet basics
  • GAE Datastore
  • Knowledge on XML-HTTP request
  • Working with Eclipse IDE

Index

This tutorial is divided into the following sections:

Code lab overview

  1. Code lab exercise is a web application template. This exercise will take you through the process of using memcache, transactions and modeling relationships.
  2. The idea of template based sample is to get hands-on experience with Google App Engine concepts. This exercise will cover all the steps required to create, deploy and monitor the application on GAE environment.
  3. The user interface is developed using HTML Java script. See the following picture to get an overview.

Setup: Download and import Code Lab Exercise 02

  1. Create the project CodeLabEx02-start as outlined in the exercise1. Make sure to create the package name as com.google.appengine.codelab. Make sure to import the template source for CodeLabEx02-start folder.

Working with the exercise

Note: Follow the instructions provided in CodeLabEx1 to create Product and Customer entities. This exercise will assume Product (eg: Mobile) and Customer (eg: John) entities present in datastore and continues further.

Using Memcache

Memory cache is used in web applications to speed up the performance by caching frequently used entities. Memcache may also be useful for caching temporary values, not backed up persistent store. The app can provide an expiration time when a value is stored. By default, the values get evicted due to memory pressure, and the least recently used get evicted first. Applications should put logic to retrieve data on cache misses.

We will upgrade Util class to using memcache feature.

  1. Modify the following lines to the Util class to get App Engine’s Memory Cache Service
  2. private static MemcacheService keycache = MemcacheServiceFactory.getMemcacheService();
  3. Add the following lines to addToCache() method at the end.
  4. keycache.put(key, entity);
  5. Add the following lines to deleteFromCache(Key key) method
  6. keycache.delete(key);
  7. Replace the return statement with the following line in getFromCache() method
  8. return (Entity) keycache.get(key);
  9. The cache methods are used for searching an entity based on the key property. We add the function getFromCache() in the utility method which is used to search for an entity. Add the following lines in the findEntity() method in Util.java. Replace the return statement.
  10. try {
      Entity entity = getFromCache(key);
      if (entity != null) {
        return entity;
      }
      return datastore.get(key);
    } catch (EntityNotFoundException ex) {
      return null;
    }

Handling transactions

With the App Engine datastore, every attempt to create, update or delete an entity happens in a transaction. A transaction ensures that every change made to the entity is saved to the datastore, or in the case of failure, none of the changes are made. This ensures consistency of data within the entity. Multiple actions can be performed on an entity within a single transaction using the transaction API. Changes to multiple entities within a transaction are done using entity groups.

We will add transaction feature to the application. Add the following lines in persistEntity() method in Util.java to enable transactions.

Key key = entity.getKey();
Transaction txn = datastore.beginTransaction();
try {
  datastore.put(entity);
  txn.commit();
} finally {
  if (txn.isActive()) {
    txn.rollback();
  } else {
    addToCache(key, entity);
  }
}

Observe that, the entities are added to the cache on successful transaction.

Modeling relationships

The following are the model entities in this exercise:

  • Product – Models the type of Product. This is more like a category and has a description. Product has one to many relationship with Item
  • Item – Model of an item. Item has many to one relationship with Product. Item creation requires the Product. To show the relationship, we add the product as a parent to the item. We also set the product name as an attribute in the item which acts as a foreign key to product entity.
  • Customer – Customer, to whom the order will be paced.
  • Order – This is an order for the Customer. Order has one to many relationship with LineItem and Many to one relationship with the Customer. As seen in item, here also we add the Customer as parent to the order and also set the customer name as an attribute to order entity to maintain the relationship.
  • Line Item – Every order has multiple lines. LineItem cannot exist alone. When an order is deleted, line item has to be deleted. This is called cascade delete. Here also, we add order as the parent for Line Item and thus create a hierarchy entity group of Customer – Order – LineItem. Every line item has an item associated with it. Because there can be only one parent for an entity, we use the Item name as an attribute.

In order to create a relationship, we will use

entity.setProperty(java.lang.String propertyName, java.lang.Object Key)
  1. entity – entity participating in the relationship property
  2. propertyName – foreign-key, as a property
  3. Key – entity key, that is part of the relation
  1. Create many to one relationshipOpen com.google.appengine.codelab.Item and replace return statement with the following code into the method createOrUpdateItem().
    Entity product = Product.getProduct(productName);
    Entity item = getSingleItem(itemName);
    if (item == null) {
      item = new Entity("Item", product.getKey());
      item.setProperty("name", itemName);
      item.setProperty("product", productName);
      item.setProperty("price", price);
      } else {
        if (price != null && !"".equals(price)) {
          item.setProperty("price", price);
        }
        if (productName != null && !"".equals(productName)) {
          item.setProperty("product", productName);
        }
      }
    Util.persistEntity(item);
    return item;

    Notice that setProperty method takes the property value and foreign-key. In this example, the product name is the foreign-key, selected from the drop down list in the UI.

    Since Item has many to one relationship with Product, deleting the Item would not affect the Product.

    Similarly, order has many to one relationship with Customer. We use the following statements in order creation to make customer as a parent to the order and also use it as an attribute. We Will add this code in the next section.

    Entity customer = Customer.getSingleCustomer(customerName);
    Key customerkey = customer.getKey();
    Entity order = new Entity("Order", customerkey);
    order.setProperty("customerName", customerName);
  2. Creating one to many relationshipAn Order has more than one line item. Order has one to many relationships with LineItem. For simplicity, in our example we have used only one LineItem per Order.

    As, the line items and orders must be saved at the same time, i.e. if any one of the entities is not saved, we have to delete all entities. So, we shall include a transaction here while saving the order. Add the following lines of code in createOrUpdateOrder() method of Order.java.

    Transaction txn = Util.getDatastoreServiceInstance().beginTransaction();
    try {
      Entity customer = Customer.getSingleCustomer(customerName);
      Key customerkey = customer.getKey();
    
      Entity order = new Entity("Order", customerkey);
      order.setProperty("customerName", customerName);
      order.setProperty("status", "Processed");
      order.setProperty("shipTo", shipTo);
      order.setProperty("city", city);
      order.setProperty("state", state);
      order.setProperty("zip", zip);
      Util.getDatastoreServiceInstance().put(order);
    
      Entity lineItem = new Entity("LineItem", order.getKey());
      // key to string can be inserted instead of name, a better option
      lineItem.setProperty("itemName", itemName);
      lineItem.setProperty("quantity", quantity);
      lineItem.setProperty("price", price);
      lineItem.setProperty("orderID", String.valueOf(order.getKey().getId()));
      Util.getDatastoreServiceInstance().put(lineItem);
      txn.commit();
    } catch (Exception e) {
      logger.log(Info.WARNING, e.getMessage());
    } finally {
      if (txn.isActive()) {
        txn.rollback();
      }
    }

    Notice that, Line Item has one to one relationship with the Item. It has foreign-key to Order as well. When we order, we can have one or more than one item that can be ordered and each item can have different quantities to be shipped. Hence, we include Line Item for every order.

  3. Deleting in one to many relationship – Cascade DeleteDeleting on order should delete the LineItems also and it must be done in single transaction. As we have customer as parent for every order, we should create a key with parent key in it and then delete those records. Here, using the customerName and Order Id we will create the whole key of order and then also find the respective line item keys. We first delete the line items and then the order in a single transaction.

    Add the following lines of code in doDelete() method of OrderServlet.java.

    String orderId = req.getParameter("id");
    String customerName = req.getParameter("parentid");
    
    Transaction txn = datastore.beginTransaction();
    try {
      Key parentKey = KeyFactory.createKey("Customer", customerName);
      Key key = KeyFactory.createKey(parentKey, "Order", Integer.parseInt(orderId));
    
      // CASCADE_ON_DELETE
      Iterable<Entity> entities = Util.listChildKeys("LineItem", key);
      final List<Key> keys = new ArrayList<Key>();
      for (Entity e : entities) {
        keys.add(e.getKey());
      }
      Util.deleteEntity(keys);
      datastore.delete(key);
      txn.commit();
    } finally {
      if (txn.isActive()) {
        txn.rollback();
      }
    }

Run the Application

  1. Refer Appendix AOnce the application is deployed, open a browser and access the following URLhttp://localhost:8888/

    The home screen is shown below.

    Create a product. The create product screen is shown below.

  2. Similarly, create an item for sale and a customer. Once these are created, go ahead and place an order for a specific customer and an item.
  3. The UI for placing the order is shown below.

    If order creation is successful, you can see the order in the list displayed.

Deploy your application to GAE

  1. Refer Appendix B

Summary

  1. Congratulations! You’ve completed the code lab exercise. You can also see the solution to the exercise in folder CodeLabEx02 folder

GAE: Tutorial: Code Lab Exercise 1: Working with Entities

Goal

This tutorial guides you through the process of creating a sample application to perform CRUD operations on entities. These entities are stored and retrieved using the DataStore APIs from datastore defined by GAE.

For this exercise we will build a sample product catalog with two entities: Product and Item. The application provides UI to perform all the CRUD operations on the entities.

Following actions will be covered in this exercise:

  1. Create entities of type product, item
  2. List entity details
  3. Edit entity details
  4. Delete entity

Time for completion

Approximately 30 minutes.

Prerequisites

Before you begin this tutorial, you should be familiar with:

  • Web application basics and deployment process
  • Servlet basics
  • Knowledge on XML-HTTP request
  • Working with Eclipse IDE

Index

This tutorial is organized into the following sections:

Code lab overview

  1. Code lab exercise is a web application template. The idea of the template based sample is to get hands-on experience with Google App Engine concepts. This exercise will cover all the steps required to create, deploy and monitor application on GAE environment.

  1. The picture shows a solution overview. Goal of this exercise is to demonstrate the GoogleDataStore service API for persisting data

    In this exercise we will create the Product and Item entities, store and retrieve them. The datastore is the service provided by the App Engine. UI is built using HTML and javascript and data from the application is sent to browser in JSON format.

    The use of memcache is demonstrated in the next exercise.

Setup : Import Code Lab Exercise 1

  1. Import the source code from the downloaded folder CodeLabEx1-starter as follows. Select File > Import from Eclipse IDE and select CodeLabEx1-starter folder.

Working with the exercise

The App Engine provides a robust and scalable Datastore for storing, searching and retrieving data. The App Engine datastore saves data objects as entities. An entity has one or more properties, named values of one of several supported data types for instance a string, an integer, or even a reference to another entity.

About Entity

An entity is an object representation of a datastore row. Unlike a row in a relational database, there are no predefined columns. There are no tables corresponding to types of data.

Entities don’t have columns – they have properties. A property represents a field of your entity. For instance, a person entity would have a kind of person, a key corresponding to a unique identifier corresponding to their name. The Kind of the entity is stored as part of the entity key. Entity Key is the primary way by which entities are fetched, even when the queries are issued. It’s similar to a primary key in an RDBMS. The key encodes your application ID, your entity’s kind, any parent entities and other metadata. The entity ID can also be a string that can be provided by the application.

This exercise shows how to create and retrieve entries to and from a data store. Each entity also has a key that uniquely identifies the entity. The simplest way is to have the Kind and Entity ID provided by the datastore. Kind indexes the entity and helps in querying the datastore efficiently.

Getting the Handle to DataStoreService

Take a look at the com.google.appengine.codelab.Util class. The Util class has all the methods to store and retrieve entities from the datastore. It is wrapper for DatastoreService provided by the App Engine api.

Edit the Util class as follows. Locate the following line of code in the Util class.

private static DatastoreService datastore = null;

Replace the above line with the code below.

private static DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
  • The getDatastoreService() method of DatastoreServiceFactory returns DataStoreService that an application can use. The DatastoreService has put()get() and delete() methods. There is no need to worry about opening or closing service.
  • persistEntity() method saves the entity using the datastore service put() API.
  • deleteEntity() method deletes the entiry using the datastore service delete() API.
  • findEntity() method searches the datastore given a key usign the datastore service find() API.
  • listEntities() methods allows to search by adding filters through Query class. Executes the query using prepare. Prepare also returns an iterator.
  • writeJSON() method is an utility method to send back the servlet response as JSON to the browser.
  1. Creating OR Updating Item Entity

    The code demonstrates creation of entities.
    An entity can be created with an app-assigned key name (in this case, name) string or a system-assigned numeric ID as its identifier to create the object. To set a key name, provide it as the second argument to the Entity constructor. Simply omit the argument, to indicate that the datastore should assign a unique numeric ID automatically.

    Entity does not contain setKey() or setKind() methods. This is because an entity’s key is immutable. Once an Entity has a key, it can never be changed.

    Locate the createOrUpdateItem() method in com.google.appengine.codelab.Item and insert the following lines of code.
    Make sure to return item instead of null.

    Entity product = Product.getProduct(productName);
    Entity item = getSingleItem(itemName);
    if(item == null){
      item = new Entity("Item",product.getKey());
      item.setProperty("name", itemName);
      item.setProperty("product", productName);
      item.setProperty("price", price);
    } else{
      if (price != null && !"".equals(price)) {
        item.setProperty("price", price);
      }          
    }
    Util.persistEntity(item);
    return item;

    util.persistEntity() invokes the DatastoreService’s put() to persist the Entity.
    The product’s name is used as key. The same key is used to retrieve the product later.

  2. Deleting Item Entity

    An Entity can be deleted using deleteItem() method in Item class. Add the below code to deleteItem() method.

    Entity entity = getSingleItem(itemKey);    
    if(entity != null){
      Util.deleteEntity(entity.getKey());
      return("Item deleted successfully.");
    } else
      return("Item not found");

    Util.deleteEntity() invokes the DatastoreService’s delete() to delete the Entity.

  3. Finding Item Entity

    To search an entity, use findEntity() method of Util.java . Add the below code to getSingleItem() method of Item class.
    Ensure the found Entity is returned.

    Query query = new Query("Item");
    query.addFilter("name", FilterOperator.EQUAL, itemName);
    List<Entity> results = Util.getDatastoreServiceInstance().prepare(query).asList(FetchOptions.Builder.withDefaults());
    if (!results.isEmpty()) {
      return (Entity)results.remove(0);
    }
    return null;

Run the Application

    Refer Appendix A 

  1. Once the application is deployed locally, open a browser and access the following URLhttp://localhost:8888/

    The welcome screen is shown below.

  2. Click on “Add” button from top panel. Make sure you are on Product tab.Fill in the create form and click on “Save” button

  3. On saving the product, the product list view appears
  4. Click “Add” button from the top panel. Make sure you are on Create Item tab.Fill the form, select the product type and then click on “Save” button.

  5. On saving the Item, the Item list view appears
  6. To update an item, Click on “Edit” which shows the below screen.
  7. To verify the entity creation, check for newly created item in the item listing.In next section, we can see the admin console and verify the data entities created.

Admin Console

  1. Login to ‘Admin Console’ to verify the entity present in the data store.Admin console can be opened on development machine using http://localhost:8888/_ah/admin

    Click on DataStore Viewer –> Select Item from Entity Kind list and click List Entities to view the product created.

Deploy your application to GAE

  1. Refer Appendix B

Summary

  1. Congratulations! You’ve completed the code lab exercise.You can also see the solution to the exercise in folder CodeLabEx1 folder.

GAE: Almacenando datos

Storing Data

The App Engine environment provides a range of options for storing your data:

  • App Engine Datastore provides a NoSQL schemaless object datastore, with a query engine and atomic transactions.
  • Google Cloud SQL provides a relational SQL database for your App Engine application, based on the familiar MySQL database.
  • Google Cloud Storage provides a storage service for objects and files up to terabytes in size.

This section details the API for accessing the App Engine Datastore from Java. For details of the other storage options, see the documentation for Cloud SQL and Cloud Storage.

The Java Datastore API

The App Engine Datastore is a schemaless object datastore providing robust, scalable storage for your web application, with the following features:

  • No planned downtime
  • Atomic transactions
  • High availability of reads and writes
  • Strong consistency for reads and ancestor queries
  • Eventual consistency for all other queries

The Java Datastore SDK includes implementations of the Java Data Objects (JDO) and Java Persistence API (JPA) interfaces, as well as a low-level Datastore API.

This reference describes the Java interfaces for the App Engine datastore, with an emphasis on JDO. It has the following sections:

  1. Datastore Overview
  2. Entities, Properties, and Keys
  3. Datastore Queries
  4. Datastore Indexes
  5. Datastore Transactions
  6. Structuring Data for Strong Consistency
  7. Using the Master/Slave Datastore
  8. Metadata
  9. Datastore Statistics
  10. Async Datastore API
  11. Datastore Callbacks
  12. JDO
  13. JPA
  14. Google Cloud Storage
  15. Javadoc Reference

Datastore Overview

The App Engine Datastore is a schemaless object datastore providing robust, scalable storage for your web application, with the following features:

  • No planned downtime
  • Atomic transactions
  • High availability of reads and writes
  • Strong consistency for reads and ancestor queries
  • Eventual consistency for all other queries

The Java Datastore SDK includes implementations of the Java Data Objects (JDO) and Java Persistence API (JPA) interfaces, as well as a low-level Datastore API.

App Engine’s primary data repository is the High Replication Datastore (HRD), in which data is replicated across multiple data centers using a system based on the Paxos algorithm. This provides a high level of availability for reads and writes. Most queries are eventually consistent.

A second storage option, the Master/Slave Datastore, has been deprecated in favor of the HRD as of April 4, 2012. Although Google will continue to support the Master/Slave Datastore in accordance with our terms of service, it is strongly recommended that all new applications use the HRD instead, and that existing applications using the Master/Slave Datastore migrate to the HRD.

Watch a video demonstration comparing the Master/Slave and High Replication Datastores.

The Datastore holds data objects known as entities. An entity has one or more properties, named values of one of several supported data types: for instance, a property can be a string, an integer, or a reference to another entity. Each entity is identified by its kind, which categorizes the entity for the purpose of queries, and a key that uniquely identifies it within its kind.

The Datastore can execute multiple operations in a single transaction. By definition, a transaction cannot succeed unless every one of its operations succeeds; if any of the operations fails, the transaction is automatically rolled back. This is especially useful for distributed web applications, where multiple users may be accessing or manipulating the same data at the same time.

Contents

  1. Comparison with Traditional Databases
  2. Java Datastore API
  3. Entities
    1. Kinds, Keys, and Identifiers
    2. Ancestor Paths
  4. Queries and Indexes
  5. Transactions
    1. Transactions and Entity Groups
    2. Cross-Group Transactions
  6. Datastore Writes and Data Visibility
  7. Datastore Statistics
  8. Quotas and Limits

Comparison with Traditional Databases

Unlike traditional relational databases, the App Engine Datastore uses a distributed architecture to automatically manage scaling to very large data sets. While the Datastore interface has many of the same features as traditional databases, it differs from them in the way it describes relationships between data objects. Entities of the same kind can have different properties, and different entities can have properties with the same name but different value types.

These unique characteristics imply a different way of designing and managing data to take advantage of the ability to scale automatically. In particular, the App Engine Datastore differs from a traditional relational database in the following important ways:

  • The App Engine Datastore is designed to scale, allowing applications to maintain high performance as they receive more traffic:
    • Datastore writes scale by automatically distributing data as necessary.
    • Datastore reads scale because the only queries supported are those whose performance scales with the size of the result set (as opposed to the data set). This means that a query whose result set contains 100 entities performs the same whether it searches over a hundred entities or a million. This property is the key reason some types of query are not supported.
  • Because all queries on App Engine are served by pre-built indexes, the types of query that can be executed are more restrictive than those allowed on a relational database with SQL. In particular, the following are not supported:
    • Join operations
    • Inequality filtering on multiple properties
    • Filtering of data based on results of a subquery
  • Unlike traditional relational databases, the App Engine Datastore doesn’t require entities of the same kind to have a consistent property set (although you can choose to enforce such a requirement in your own application code).

For more in-depth information about the design of the Datastore, read our series of articles on Mastering the Datastore.

Java Datastore API

The App Engine Java SDK provides a low-level Datastore API with simple operations on entities, including getputdelete, and query. You can just use this low-level API directly in your applications, or use it as a base on which to implement other interface adapters. The SDK also includes implementations of the Java Data Objects (JDO) and Java Persistence API (JPA) interfaces for modeling and persisting data. These standard interfaces include mechanisms for defining classes for data objects and for performing queries.

In addition to the standard frameworks and low-level Datastore API, the Java SDK supports other frameworks designed to simplify Datastore usage for Java developers. Many Java developers use these frameworks; the Google App Engine team highly recommends them and encourages you to investigate them.

  • Objectify is a very simple and convenient interface to the App Engine Datastore that helps you avoid some of the complexities presented by JDO/JPA and the low-level Datastore.
  • Twig is a configurable object persistence interface that improves support for inheritance, polymorphism, and generic types. Like Objectify, Twig also helps you avoid complexities posed by JDO and the low-level Datastore.
  • Slim3 is a full-stack model-view-controller framework that you can use for a wide variety of App Engine functions, including (but not limited to) the Datastore.

Entities

Objects in the App Engine Datastore are known as entities. An entity has one or more named properties, each of which can have one or more values. Property values can belong to a variety of data types, including integers, floating-point numbers, strings, dates, and binary data, among others. A query on a property with multiple values tests whether any of the values meets the query criteria. This makes such properties useful for membership testing.

Note: Datastore entities are schemaless: unlike traditional relational databases, the App Engine Datastore does not require that all entities of a given kind have the same properties or that all of an entity’s values for a given property be of the same data type. If a formal schema is needed, the application itself is responsible for ensuring that entities conform to it.

Kinds, Keys, and Identifiers

Each Datastore entity is of a particular kind, which categorizes the entity for the purpose of queries: for instance, a human resources application might represent each employee at a company with an entity of kind Employee. In addition, each entity has its own key, which uniquely identifies it. The key consists of the following components:

  • The entity’s kind
  • An identifier, which can be either
    • key name string
    • an integer numeric ID
  • An optional ancestor path locating the entity within the Datastore hierarchy

The identifier is assigned when the entity is created. Because it is part of the entity’s key, it is associated permanently with the entity and cannot be changed. It can be assigned in either of two ways:

  • Your application can specify its own key name string for the entity.
  • You can have the Datastore automatically assign the entity an integer numeric ID.

Note: Instead of using key name strings or generating numeric IDs automatically, advanced applications may sometimes wish to assign their own numeric IDs manually to the entities they create. Be aware, however, that if you choose this option you must take special steps to prevent your manually assigned numeric IDs from conflicting with those assigned automatically by the Datastore; see the Entities, Properties, and Keys page for further details.

Ancestor Paths

Entities in the Datastore form a hierarchically structured space similar to the directory structure of a file system. When you create an entity, you can optionally designate another entity as its parent; the new entity is a child of the parent entity. An entity without a parent is a root entity. The association between an entity and its parent is permanent, and cannot be changed once the entity is created. The Datastore will never assign the same numeric ID to two entities with the same parent, or to two root entities (those without a parent).

An entity’s parent, parent’s parent, and so on recursively, are its ancestors; its children, children’s children, and so on, are its descendants. An entity and its descendants are said to belong to the same entity group. The sequence of entities beginning with a root entity and proceeding from parent to child, leading to a given entity, constitute that entity’s ancestor path. The complete key identifying the entity consists of a sequence of kind-identifier pairs specifying its ancestor path and terminating with those of the entity itself:

Person:GreatGrandpa / Person:Grandpa / Person:Dad / Person:Me

For a root entity, the ancestor path is empty and the key consists solely of the entity’s own kind and identifier:

Person:GreatGrandpa

Queries and Indexes

In addition to retrieving entities from the Datastore directly by their keys, an application can perform a query to retrieve them by the values of their properties. The query operates on entities of a given kind; it can specify filters on the entities’ property values, keys, and ancestors, and can return zero or more entities as results. A query can also specify sort orders to sequence the results by their property values. The results include all entities that have at least one (possibly null) value for every property named in the filters and sort orders, and whose property values meet all the specified filter criteria. The query can return entire entities, projected entities, or just entity keys.

A typical query includes the following:

  • An entity kind to which the query applies
  • Zero or more filters based on the entities’ property values, keys, and ancestors
  • Zero or more sort orders to sequence the results

When executed, the query retrieves all entities of the given kind that satisfy all of the given filters, sorted in the specified order.

Note: To conserve memory and improve performance, a query should, whenever possible, specify a limit on the number of results returned.

A query can also include an ancestor filter limiting the results to just the entity group descended from a specified ancestor. Such a query is known as an ancestor query. By default, ancestor queries return strongly consistent results, which are guaranteed to be up to date with the latest changes to the data. Non-ancestor queries, by contrast, can span the entire Datastore rather than just a single entity group, but are only eventually consistent and may return stale results. If strong consistency is important to your application, you may need to take this into account when structuring your data, placing related entities in the same entity group so they can be retrieved with an ancestor rather than a non-ancestor query; see Structuring Data for Strong Consistency for more information.

Note: Consistency considerations are a bit different with the Master/Slave Datastore; see Using the Master/Slave Datastore for details.

Every Datastore query computes its results using one or more indexes, tables containing entities in a sequence specified by the index’s properties and, optionally, the entity’s ancestors. The indexes are updated incrementally to reflect any changes the application makes to its entities, so that the correct results of all queries are immediately available with no further computation needed.

App Engine predefines a simple index on each property of an entity. An App Engine application can define further custom indexes in an index configuration file named datastore-indexes.xml. The development web server automatically adds suggestions to this file as it encounters queries that cannot be executed with the existing indexes. You can tune indexes manually by editing the file before uploading the application.

Note: This index-based query mechanism supports a wide range of queries and is suitable for most applications. However, it does not support some kinds of query common in other database technologies: in particular, joins and aggregate queries aren’t supported within the Datastore query engine. See the Datastore Queries page for limitations on App Engine Datastore queries.

Transactions

Every attempt to insert, update, or delete an entity takes place in the context of a transaction. A single transaction can include any number of such operations. To maintain the consistency of the data, the transaction ensures that all of the operations it contains are applied to the Datastore as a unit or, if any of the operations fails, that none of them are applied.

You can perform multiple actions on an entity within a single transaction. For example, to increment a counter field in an object, you need to read the value of the counter, calculate the new value, and then store it back. Without a transaction, it is possible for another process to increment the counter between the time you read the value and the time you update it, causing your application to overwrite the updated value. Doing the read, calculation, and write in a single transaction ensures that no other process can interfere with the increment.

Transactions and Entity Groups

Only ancestor queries are allowed within a transaction: that is, each query must be limited to a single entity group. The transaction itself can apply to multiple entities, which can belong either to a single entity group or (in the case of a cross-group transaction) to as many as five different entity groups.

The Datastore uses optimistic concurrency to manage transactions. When two or more application instances try to change the same entity group at the same time (either updating existing entities or creating new ones), the first application to commit its changes will succeed and all others will fail on commit. These other applications can then try their transactions again to apply them to the updated data. Note that this limits the number of concurrent writes you can do to any entity in a given entity group.

Cross-Group Transactions

A transaction on entities belonging to different entity groups is called a cross-group (XG) transaction. The transaction can be applied across a maximum of five entity groups, and will succeed as long as no concurrent transaction touches any of the entity groups to which it applies. This gives you more flexibility in organizing your data, because you aren’t forced to put disparate pieces of data under the same ancestor just to perform atomic writes on them.

As in a single-group transaction, you cannot perform a non-ancestor query in an XG transaction. You can, however, perform ancestor queries on separate entity groups. Nontransactional (non-ancestor) queries may see all, some, or none of the results of a previously committed transaction. (For background on this issue, see Datastore Writes and Data Visibility.) However, such nontransactional queries are more likely to see the results of a partially committed XG transaction than those of a partially commited single-group transaction.

Note: The first read of an entity group in an XG transaction may throw a ConcurrentModificationException if there is a conflict with other transactions accessing that same entity group. This means that even an XG transaction that performs only reads can fail with a concurrency exception.

An XG transaction that touches only a single entity group has exactly the same performance and cost as a single-group, non-XG transaction. In an XG transaction that touches multiple entity groups, operations cost the same as if they were performed in a non-XG transaction, but may experience higher latency.

Datastore Writes and Data Visibility

This section describes the behavior of App Engine with the High Replication Datastore (HRD). The deprecated Master/Slave Datastore behaves differently in a few ways; see the Using the Master/Slave Datastore page for details. For a fuller discussion of this topic, see the articles Life of a Datastore Write and Transaction Isolation in App Engine.

Data is written to the Datastore in two phases:

  1. In the Commit phase, the entity data is recorded in a log.
  2. The Apply phase consists of two actions performed in parallel:
    • The entity data is written.
    • The index rows for the entity are written. (Note that this can take longer than writing the data itself.)

The write operation returns immediately after the Commit phase and the Apply phase then takes place asynchronously. If a failure occurs during the Commit phase, there are automatic retries; but if failures continue, the Datastore returns an error message that your application receives as an exception. If the Commit phase succeeds but the Apply fails, the Apply is rolled forward to completion when one of the following occurs:

  • Periodic Datastore sweeps check for uncompleted Commit jobs and apply them.
  • Certain application operations (getputdelete, and ancestor queries) that use the affected entity group cause any changes that have been committed but not yet applied to be completed before proceeding with the new operation.

This write behavior can have several implications for how and when data is visible to your application at different parts of the Commit and Apply phases:

  • If a write operation reports a timeout error, it cannot be determined (without attempting to read the data) whether the operation succeeded or failed.
  • Because Datastore gets and ancestor queries apply any outstanding modifications before executing, these operations always see a consistent view of all previous successful transactions. This means that a get operation (looking up an updated entity by its key) is guaranteed to see the latest version of that entity.
  • As long as a few hundred milliseconds may elapse from the time a write operation returns until the transaction is completely applied. In this case, queries spanning more than one entity group cannot determine whether there are any outstanding modifications before executing and may return stale results.
  • The timing of concurrent query requests may affect their results. If an entity initially satisfies a query but is later changed so that it no longer does, the entity may still be included in the query’s result set; it will be omitted only if the query executes after the Apply phase of the update has been completed (that is, after the indexes have been written).

Datastore Statistics

The Datastore maintains statistics about the data stored for an application, such as how many entities there are of a given kind or how much space is used by property values of a given type. You can view these statistics in the Administration Console under DatastoreStatistics. You can also use the Datastore API to access these values programmatically from within the application by querying for specially named entities; see Datastore Statistics in Java for more information.

Quotas and Limits

Various aspects of your application’s Datastore usage are counted toward your App Engine resource quotas:

  • Each call to the Datastore API counts toward the Datastore API Calls quota. (Note that some library calls result in multiple calls to the API, and so use more of your quota.)
  • Data sent to the Datastore by the application counts toward the Data Sent to Datastore API quota.
  • Data received by the application from the Datastore counts toward the Data Received from Datastore API quota.
  • The total amount of data currently stored in the Datastore for the application cannot exceed the Stored Data (billable) quota. This includes all entity properties and keys, as well as the indexes needed to support querying those entities. See the article How Entities and Indexes Are Stored for a complete breakdown of the metadata required to store entities and indexes at the Bigtable level.

For information on systemwide safety limits, see the Quotas and Limits page and the Quota Details section of the Administration Console. In addition to such systemwide limits, the following limits apply specifically to the use of the Datastore:

Limit Amount
Maximum entity size 1 megabyte
Maximum transaction size 10 megabytes
Maximum number of index entries for an entity 20000
Maximum number of bytes in composite indexes for an entity 2 megabytes

GAE: Primeros pasos

[Fuente: https://developers.google.com/appengine/docs/java/gettingstarted/]

This tutorial describes how to develop and deploy a simple Java project with Google App Engine. The example project, a guest book, demonstrates how to use the Java runtime environment, and how to use several App Engine services, including the datastore and Google Accounts.

Introducción

Welcome to Google App Engine! Creating an App Engine application is easy, and only takes a few minutes. And it’s free to start: upload your app and share it with users right away, at no charge and with no commitment required.

Google App Engine applications can currently be written in Java, Python, or Go. This tutorial covers Java. If you’re more likely to use Python to build your applications, see Getting Started: Python. If you want to use Go, see Getting Started: Go.

In this tutorial, you will learn how to:

  • build an App Engine application using standard Java web technologies, such as servlets and JSPs
  • create an App Engine Java project with Eclipse, and without
  • use the Google Plugin for Eclipse for App Engine development
  • use the App Engine datastore with the Datastore API
  • integrate an App Engine application with Google Accounts for user authentication
  • upload your app to App Engine

By the end of the tutorial, you will have implemented a working application, a simple guest book that lets users post messages to a public message board.

Installing the Java SDK

You develop and upload Java applications for Google App Engine using the App Engine Java software development kit (SDK).

The SDK includes software for a web server that you can run on your own computer to test your Java applications. The server simulates all of the App Engine services, including a local version of the datastore, Google Accounts, and the ability to fetch URLs and send email from your computer using the App Engine APIs.

Getting Java

Google App Engine supports Java 5 and Java 6. When your Java application is running on App Engine, it runs using the Java 6 virtual machine (JVM) and standard libraries. Ideally, you should use Java 6 for compiling and testing your application to ensure that the local server behaves similarly to App Engine.

For developers that don’t have easy access to Java 6 (such as developers using Mac OS X), the App Engine SDK is compatible with Java 5. You can upload compiled classes and JARs made with Java 5 to App Engine.

If necessary, download and install the Java SE Development Kit (JDK) for your platform. Mac users, see Apple’s Java developer site to download and install the latest version of the Java Developer Kit available for Mac OS X.

Once the JDK is installed, run the following commands from a command prompt (for Windows, Command Prompt; for Mac OS X, Terminal) to verify that you can run the commands, and to determine which version is installed. If you have Java 6 installed, these commands will report a version number similar to 1.6.0. If you have Java 5 installed, the version number will be similar to 1.5.0.

java -version

javac -version

Using Eclipse and the Google Plugin for Eclipse

If you are using the Eclipse development environment, the easiest way to develop, test and upload App Engine apps is to use the Google Plugin for Eclipse. The plugin includes everything you need to build, test and deploy your app, entirely within Eclipse.

The plugin is available for Eclipse versions 3.3, 3.4, 3.5, 3.6, and 3.7. You can install the plugin using the Software Update feature of Eclipse. The installation locations are as follows:

  • The Google Plugin for Eclipse, for Eclipse 3.3 (Europa):
    https://dl.google.com/eclipse/plugin/3.3
  • The Google Plugin for Eclipse, for Eclipse 3.4 (Ganymede):
    https://dl.google.com/eclipse/plugin/3.4
  • The Google Plugin for Eclipse, for Eclipse 3.5 (Galileo):
    https://dl.google.com/eclipse/plugin/3.5
  • The Google Plugin for Eclipse, for Eclipse 3.6 (Helios):
    https://dl.google.com/eclipse/plugin/3.6
  • The Google Plugin for Eclipse, for Eclipse 3.7 (Indigo):
    https://dl.google.com/eclipse/plugin/3.7

For details on how to use Software Update to install the plugin, and how to create a new project, see Using the Google Eclipse Plugin.

Using Apache Ant

If you want to compile and run your application using the command line, or an IDE other than Eclipse, you will need to install something to manage this process. Apache Ant is one such solution. Full directions on installing and setting up Apache Ant to work with App Engine can be found at Using Apache Ant.

Getting the SDK

If you are using Eclipse and the Google Plugin, you can install the App Engine SDK from Eclipse using Software Update. If you haven’t already, install the “Google App Engine Java SDK” component using the locations above.

If you are not using Eclipse or the Google Plugin, you can download the App Engine Java SDK as a Zip archive.

Download the App Engine Java SDK. Unpack the archive in a convenient location on your hard drive.

Note: Unpacking the archive creates a directory whose name is something like appengine-java-sdk-X.X.X, where X.X.X is the SDK version number. Throughout this documentation, this directory will be referred to as appengine-java-sdk/. You may want to rename the directory after unpacking.

Trying a Demo Application

The App Engine Java SDK includes several demo applications in the demos/ directory. The final version of the guest book application you will create in this tutorial is included under the directory guestbook/. This demo has been precompiled for you so you can try it right away.

If you are using Eclipse, the SDK is located in your Eclipse installation directory, under plugins/com.google.appengine.eclipse.sdkbundle_VERSION/, where VERSION is a version identifier for the SDK. From the command line, change the current working directory to this directory to run the following command. If you’re using Mac OS X or Linux, you may need to give the command files executable permissions before you can run them (such as with the command chmod u+x dev_appserver.sh).

If you are using Windows, start the guest book demo in the development server by running the following command at a command prompt:

appengine-java-sdkbindev_appserver.cmd appengine-java-sdkdemosguestbookwar

If you are using Mac OS X or Linux, run the following command:

./appengine-java-sdk/bin/dev_appserver.sh appengine-java-sdk/demos/guestbook/war

The development server starts, and listens for requests on port 8080. Visit the following URL in your browser:

Note: When you start the development server from within Eclipse using the Google Plugin for Eclipse (discussed later), the server uses the port 8888 by default:http://localhost:8888/

For more information about running the development web server from the command line, including how to change which port it uses, see the Dev Web Server reference.

To stop the server, make sure the command prompt window is active, then press Control-C.

Creating a Project

App Engine Java applications use the Java Servlet standard for interacting with the web server environment. An application’s files, including compiled classes, JARs, static files and configuration files, are arranged in a directory structure using the WAR standard layout for Java web applications. You can use any development process you like to develop web servlets and produce a WAR directory. (WAR archive files are not yet supported by the SDK.)

The Project Directory

For this tutorial, we will use a single directory named Guestbook/ for all project files. A subdirectory named src/ contains the Java source code, and a subdirectory named war/contains the complete application arranged in the WAR format. Our build process compiles the Java source files and puts the compiled classes in the appropriate location in war/.

The complete project directory looks like this:

Guestbook/
  src/
    ...Java source code...
    META-INF/
      ...other configuration...
  war/
    ...JSPs, images, data files...
    WEB-INF/
      ...app configuration...
      lib/
        ...JARs for libraries...
      classes/
        ...compiled classes...

If you are using Eclipse, create a new project by clicking the New Web Application Project button in the toolbarThe New Web Application Project button. Give the project a “Project name” of Guestbook and a “Package” of guestbook. Uncheck “Use Google Web Toolkit,” and ensure “Use Google App Engine” is checked. See Using the Google Plugin for Eclipse for more information. The wizard creates the directory structure, and the files described below.

If you are not using Eclipse, create the directory structure described above. As you read each of the files described in this section, create the files using the given locations and names.

You can also copy the new project template included with the SDK, in the appengine-java-sdk/demos/new_project_template/ directory.

The Servlet Class

App Engine Java applications use the Java Servlet API to interact with the web server. An HTTP servlet is an application class that can process and respond to web requests. This class extends either the javax.servlet.GenericServlet class or the javax.servlet.http.HttpServlet class.

Our guest book project begins with one servlet class, a simple servlet that displays a message.

If you are not using the Eclipse plugin, create the directories for the path src/guestbook/, then create the servlet class file described below.

In the directory src/guestbook/, make a file named GuestbookServlet.java with the following contents:

package guestbook;

import java.io.IOException;
import javax.servlet.http.*;

public class GuestbookServlet extends HttpServlet {
    public void doGet(HttpServletRequest req, HttpServletResponse resp)
            throws IOException {
        resp.setContentType("text/plain");
        resp.getWriter().println("Hello, world");
    }
}

The web.xml File

When the web server receives a request, it determines which servlet class to call using a configuration file known as the “web application deployment descriptor.” This file is named web.xml, and resides in the war/WEB-INF/ directory in the WAR. WEB-INF/ and web.xml are part of the servlet specification.

In the directory war/WEB-INF/, a file named web.xml has the following contents:

<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE web-app PUBLIC
 "-//Oracle Corporation//DTD Web Application 2.3//EN"
 "http://java.sun.com/dtd/web-app_2_3.dtd">

<web-app xmlns="http://java.sun.com/xml/ns/javaee" version="2.5">
    <servlet>
        <servlet-name>guestbook</servlet-name>
        <servlet-class>guestbook.GuestbookServlet</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>guestbook</servlet-name>
        <url-pattern>/guestbook</url-pattern>
    </servlet-mapping>
    <welcome-file-list>
        <welcome-file>index.html</welcome-file>
    </welcome-file-list>
</web-app>

This web.xml file declares a servlet named guestbook, and maps it to the URL path /guestbook. It also says that, whenever the user fetches a URL path that is not already mapped to a servlet and represents a directory path inside the application’s WAR, the server should check for a file named index.html in that directory and serve it if found.

The appengine-web.xml File

App Engine needs one additional configuration file to figure out how to deploy and run the application. This file is named appengine-web.xml, and resides in WEB-INF/alongside web.xml. It includes the registered ID of your application (Eclipse creates this with an empty ID for you to fill in later), the version number of your application, and lists of files that ought to be treated as static files (such as images and CSS) and resource files (such as JSPs and other application data).

In the directory war/WEB-INF/, a file named appengine-web.xml has the following contents:

<?xml version="1.0" encoding="utf-8"?>
<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
    <application></application>
    <version>1</version>
</appengine-web-app>

appengine-web.xml is specific to App Engine, and is not part of the servlet standard. You can find XML schema files describing the format of this file in the SDK, in the appengine-java-sdk/docs/ directory. See Configuring an App for more information about this file.

Running the Project

The App Engine SDK includes a web server application you can use to test your application. The server simulates the App Engine environment and services, including sandbox restrictions, the datastore, and the services.

If you are using Eclipse, you can start the development server within the Eclipse debugger. Make sure the project (“Guestbook”) is selected, then in the Run menu, select Debug As > Web Application. See Using the Google Plugin for Eclipse for details on creating the debug configuration.

If you are not using Eclipse, see Using Apache Ant for a build script that can build the project and start the development server. To start the server with this build script, enter the following command: ant runserver To stop the server, hit Control-C.

Testing the Application

Start the server, then visit the server’s URL in your browser. If you’re using Eclipse and the Google Eclipse plugin, the server runs using port 8888 by default:

If you’re using the dev_appserver command to start the server, the default port is 8080:

For the rest of this tutorial, we’ll assume the server is using port 8888.

The server calls the servlet, and displays the message in the browser.

Using the Users Service

Google App Engine provides several useful services based on Google infrastructure, accessible by applications using libraries included with the SDK. One such service is the Users service, which lets your application integrate with Google user accounts. With the Users service, your users can use the Google accounts they already have to sign in to your application.

Let’s use the Users service to personalize this application’s greeting.

Using Users

Edit src/guestbook/GuestbookServlet.java as indicated to resemble the following:

package guestbook;

import java.io.IOException;
import javax.servlet.http.*;
import com.google.appengine.api.users.User;
import com.google.appengine.api.users.UserService;
import com.google.appengine.api.users.UserServiceFactory;

public class GuestbookServlet extends HttpServlet {
    public void doGet(HttpServletRequest req, HttpServletResponse resp)
              throws IOException {
        UserService userService = UserServiceFactory.getUserService();
        User user = userService.getCurrentUser();

        if (user != null) {
            resp.setContentType("text/plain");
            resp.getWriter().println("Hello, " + user.getNickname());
        } else {
            resp.sendRedirect(userService.createLoginURL(req.getRequestURI()));
        }
    }
}

If you are using Eclipse and your development server is running in the debugger, when you save your changes to this file, Eclipse compiles the new code automatically, then attempts to insert the new code into the already-running server. Changes to classes, JSPs, static files and appengine-web.xml are reflected immediately in the running server without needing to restart. If you change web.xml or other configuration files, you must stop and start the server to see the changes.

If you are using Ant, you must stop the server and rebuild the project to see changes made to source code. Changes to JSPs and static files do not require restarting the server.

Rebuild your project and restart the server, if necessary. Test the application by visiting the servlet URL in your browser:

Instead of displaying the message, the server prompts you for an email address. Enter any email address (such as alfred@example.com, then click “Log In.” The app displays a message, this time containing the email address you entered.

The new code for the GuestbookServlet class uses the Users API to check if the user is signed in with a Google Account. If not, the user is redirected to the Google Accounts sign-in screen. userService.createLoginURL(...) returns the URL of the sign-in screen. The sign-in facility knows to redirect the user back to the app by the URL passed to createLoginURL(...), which in this case is the URL of the current page.

The development server knows how to simulate the Google Accounts sign-in facility. When run on your local machine, the redirect goes to the page where you can enter any email address to simulate an account sign-in. When run on App Engine, the redirect goes to the actual Google Accounts screen.

You are now signed in to your test application. If you reload the page, the message will display again.

To allow the user to sign out, provide a link to the sign-out screen, generated by the method createLogoutURL(). Note that a sign-out link will sign the user out of all Google services.

Using JSPs

While we could output the HTML for our user interface directly from the Java servlet code, this would be difficult to maintain as the HTML gets complicated. It’s better to use a template system, with the user interface designed and implemented in separate files with placeholders and logic to insert data provided by the application. There are many template systems available for Java, any of which would work with App Engine.

For this tutorial, we’ll use JavaServer Pages (JSPs) to implement the user interface for the guest book. JSPs are part of the servlet standard. App Engine compiles JSP files in the application’s WAR automatically as one large JAR file, then maps the URL paths accordingly.

Hello, JSP!

Our guest book app writes strings to an output stream, but this could also be written as a JSP. Let’s begin by porting the latest version of the example to a JSP.

In the directory war/, create a file named guestbook.jsp with the following contents:

<%@ page contentType="text/html;charset=UTF-8" language="java" %>
<%@ page import="java.util.List" %>
<%@ page import="com.google.appengine.api.users.User" %>
<%@ page import="com.google.appengine.api.users.UserService" %>
<%@ page import="com.google.appengine.api.users.UserServiceFactory" %>
<%@ taglib prefix="fn" uri="http://java.sun.com/jsp/jstl/functions" %>

&lt;html>

  <body>

<%
    UserService userService = UserServiceFactory.getUserService();
    User user = userService.getCurrentUser();
    if (user != null) {
      pageContext.setAttribute("user", user);
%>
<p>Hello, ${fn:escapeXml(user.nickname)}! (You can
<a href="<%= userService.createLogoutURL(request.getRequestURI()) %>">sign out</a>.)</p>
<%
    } else {
%>
<p>Hello!
<a href="<%= userService.createLoginURL(request.getRequestURI()) %>">Sign in</a>
to include your name with greetings you post.</p>
<%
    }
%>

  </body>
</html>

By default, any file in war/ or a subdirectory (other than WEB-INF/) whose name ends in .jsp is automatically mapped to a URL path. The URL path is the path to the .jsp file, including the filename. This JSP will be mapped automatically to the URL /guestbook.jsp.

For the guest book app, we want this to be the application’s home page, displayed when someone accesses the URL /. An easy way to do this is to declare in web.xml that guestbook.jsp is the “welcome” servlet for that path.

Edit war/WEB-INF/web.xml and replace the current <welcome-file> element in the <welcome-file-list>. Be sure to remove index.html from the list, as static files take precedence over JSP and servlets.

    <welcome-file-list>
        <welcome-file>guestbook.jsp</welcome-file>
    </welcome-file-list>

Tip: If you are using Eclipse, the editor may open this file in “Design” mode. To edit this file as XML, select the “Source” tab at the bottom of the frame.

Stop then start the development server. Visit the following URL:

The app displays the contents of guestbook.jsp, including the user nickname if the user is signed in. We want to HTML-escape any text which users provide in case that text contains HTML. To do this for the user nickname, we use the JSP’s pageContext so that java code can “see” the string, then call the escapeXML function we imported via the taglib element.

When you upload your application to App Engine, the SDK compiles all JSPs into one JAR file, and that is what gets uploaded.

The Guestbook Form

Our guest book application will need a web form so the user can post a new greeting, and a way to process that form. The HTML of the form will go into the JSP. The destination of the form will be a new URL, /sign, to be handled by a new servlet class, SignGuestbookServletSignGuestbookServlet will process the form, then redirect the user’s browser back to /guestbook.jsp. For now, the new servlet will just write the posted message to the log.

Edit guestbook.jsp, and put the following lines just above the closing </body> tag:

  ...

  <form action="/sign" method="post">
    <div><textarea name="content" rows="3" cols="60"></textarea></div>
    <div><input type="submit" value="Post Greeting" /></div>
  </form>

  </body>
</html>

Create a new class named SignGuestbookServlet in the package guestbook. (Non-Eclipse users, create the file SignGuestbookServlet.java in the directory src/guestbook/.) Give the source file the following contents:

package guestbook;

import java.io.IOException;
import java.util.logging.Logger;
import javax.servlet.http.*;
import com.google.appengine.api.users.User;
import com.google.appengine.api.users.UserService;
import com.google.appengine.api.users.UserServiceFactory;

public class SignGuestbookServlet extends HttpServlet {
    private static final Logger log = Logger.getLogger(SignGuestbookServlet.class.getName());

    public void doPost(HttpServletRequest req, HttpServletResponse resp)
                throws IOException {
        UserService userService = UserServiceFactory.getUserService();
        User user = userService.getCurrentUser();

        String content = req.getParameter("content");
        if (content == null) {
            content = "(No greeting)";
        }
        if (user != null) {
            log.info("Greeting posted by user " + user.getNickname() + ": " + content);
        } else {
            log.info("Greeting posted anonymously: " + content);
        }
        resp.sendRedirect("/guestbook.jsp");
    }
}

Edit war/WEB-INF/web.xml and add the following lines to declare the SignGuestbookServlet servlet and map it to the /sign URL:

<web-app xmlns="http://java.sun.com/xml/ns/javaee" version="2.5">
    ...

    <servlet>
        <servlet-name>sign</servlet-name>
        <servlet-class>guestbook.SignGuestbookServlet</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>sign</servlet-name>
        <url-pattern>/sign</url-pattern>
    </servlet-mapping>

    ...
</web-app>

This new servlet uses the java.util.logging.Logger class to write messages to the log. You can control the behavior of this class using a logging.properties file, and a system property set in the app’s appengine-web.xml file. If you are using Eclipse, your app was created with a default version of this file in your app’s src/ and the appropriate system property.

If you are not using Eclipse, you must set up the Logger configuration file manually. Copy the example file from the SDK appengine-java-sdk/config/user/logging.properties to your app’s war/WEB-INF/ directory. Then edit the app’s war/WEB-INF/appengine-web.xml file as indicated:

<appengine-web-app xmlns="http://appengine.google.com/ns/1.0">
    ...

    <system-properties>
        <property name="java.util.logging.config.file" value="WEB-INF/logging.properties"/>
    </system-properties>

</appengine-web-app>

The servlet logs messages using the INFO log level (using log.info()). The default log level is WARNING, which suppresses INFO messages from the output. To change the log level for all classes in the guestbook package, edit the logging.properties file and add an entry for guestbook.level, as follows:

.level = WARNING
guestbook.level = INFO

...

Tip: When your app logs messages using the java.util.logging.Logger API while running on App Engine, App Engine records the messages and makes them available for browsing in the Admin Console, and available for downloading using the AppCfg tool. The Admin Console lets you browse messages by log level.

Rebuild and restart, then test http://localhost:8888/. The form displays. Enter some text in the form, and submit. The browser sends the form to the app, then redirects back to the empty form. The greeting data you entered is logged to the console by the server.

Using the Datastore

Storing data in a scalable web application can be tricky. A user could be interacting with any of dozens of web servers at a given time, and the user’s next request could go to a different web server than the previous request. All web servers need to be interacting with data that is also spread out across dozens of machines, possibly in different locations around the world.

With Google App Engine, you don’t have to worry about any of that. App Engine’s infrastructure takes care of all of the distribution, replication, and load balancing of data behind a simple API—and you get a powerful query engine and transactions as well.

App Engine’s data repository, the High Replication Datastore (HRD), uses the Paxos algorithm to replicate data across multiple data centers. Data is written to the Datastore in objects known as entities. Each entity has a key that uniquely identifies it. An entity can optionally designate another entity as its parent; the first entity is a child of the parent entity. The entities in the Datastore thus form a hierarchically structured space similar to the directory structure of a file system. An entity’s parent, parent’s parent, and so on recursively, are its ancestors; its children, children’s children, and so on, are its descendants. An entity without a parent is a root entity.

The Datastore is extremely resilient in the face of catastrophic failure, but its consistency guarantees may differ from what you’re familiar with. Entities descended from a common ancestor are said to belong to the same entity group; the common ancestor’s key is the group’s parent key, which serves to identify the entire group. Queries over a single entity group, called ancestor queries, refer to the parent key instead of a specific entity’s key. Entity groups are a unit of both consistency and transactionality: whereas queries over multiple entity groups may return stale, eventually consistent results, those limited to a single entity group always return up-to-date, strongly consistent results.

The code samples in this guide organize related entities into entity groups, and use ancestor queries on those entity groups to return strongly consistent results. In the example code comments, we highlight some ways this might affect the design of your application. For more detailed information, see Structuring Data for Strong Consistency.

Note: If you built your application using an earlier version of this Getting Started Guide, please note that the sample application has changed. You can still find the sample code for the original Guestbook application, which does not use ancestor queries, in the demos directory of the SDK.

The Datastore is one of several App Engine services offering a choice of standards-based or low-level APIs. The standards-based APIs decouple your application from the underlying App Engine services, making it easier to port your application to other hosting environments and other database technologies, if you ever need to. The low-level APIs expose the service’s capabilities directly; you can use them as a base on which to implement new adapter interfaces, or just use them directly in your application.

App Engine includes support for two different API standards for the Datastore: Java Data Objects (JDO) and the Java Persistence API (JPA). These interfaces are provided by DataNucleus Access Platform, an open-source implementation of several Java persistence standards, with an adapter for the App Engine Datastore.

For clarity getting started, we’ll use the low-level API to retrieve and post messages left by users.

Updating Our Servlet to Store Data

Here is an updated version of src/SignGuestbookServlet.java that stores greetings in the Datastore. We will discuss the changes made here below.

package guestbook;

import com.google.appengine.api.datastore.DatastoreService;
import com.google.appengine.api.datastore.DatastoreServiceFactory;
import com.google.appengine.api.datastore.Entity;
import com.google.appengine.api.datastore.Key;
import com.google.appengine.api.datastore.KeyFactory;
import com.google.appengine.api.users.User;
import com.google.appengine.api.users.UserService;
import com.google.appengine.api.users.UserServiceFactory;

import java.io.IOException;
import java.util.Date;

import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

public class SignGuestbookServlet extends HttpServlet {
    public void doPost(HttpServletRequest req, HttpServletResponse resp)
                throws IOException {
        UserService userService = UserServiceFactory.getUserService();
        User user = userService.getCurrentUser();

        // We have one entity group per Guestbook with all Greetings residing
        // in the same entity group as the Guestbook to which they belong.
        // This lets us run a transactional ancestor query to retrieve all
        // Greetings for a given Guestbook.  However, the write rate to each
        // Guestbook should be limited to ~1/second.
        String guestbookName = req.getParameter("guestbookName");
        Key guestbookKey = KeyFactory.createKey("Guestbook", guestbookName);
        String content = req.getParameter("content");
        Date date = new Date();
        Entity greeting = new Entity("Greeting", guestbookKey);
        greeting.setProperty("user", user);
        greeting.setProperty("date", date);
        greeting.setProperty("content", content);

        DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
        datastore.put(greeting);

        resp.sendRedirect("/guestbook.jsp?guestbookName=" + guestbookName);
    }
}

Storing the Submitted Greetings

The low-level Datastore API for Java provides a schemaless interface for creating and storing entities. The low-level API does not require entities of the same kind to have the same properties, nor for a given property to have the same type for different entities. The following code snippet constructs the Greeting entity in the same entity group as the guestbook to which it belongs:

        Entity greeting = new Entity("Greeting", guestbookKey);
        greeting.setProperty("user", user);
        greeting.setProperty("date", date);
        greeting.setProperty("content", content);

In our example, each Greeting has the posted content, and also stores the user information about who posted, and the date on which the post was submitted. When initializing the entity, we supply the entity name, Greeting, as well as a guestbookKey argument that sets the parent of the entity we are storing. Objects in the Datastore that share a common ancestor belong to the same entity group.

After we construct the entity, we instantiate the Datastore service, and put the entity in the Datastore:

        DatastoreService datastore =
                DatastoreServiceFactory.getDatastoreService();
        datastore.put(greeting);

Because querying is strongly consistent only within entity groups, we assign all Greetings to the same entity group by setting the same parent for each Greeting. This means a user will always see a Greeting immediately after it was written. However, the rate at which you can write to the same entity group is limited to 1 write to the entity group per second. When you design a real application, you’ll need to keep this fact in mind. Note that by using services such as Memcache, you can mitigate the chance that a user won’t see fresh results when querying across entity groups immediately after a write.

Updating the JSP

We also need to modify the JSP we wrote earlier to display Greetings from the Datastore, and also include a form for submitting Greetings. Here is our updated guestbook.jsp:

<%@ page contentType="text/html;charset=UTF-8" language="java" %>
<%@ page import="java.util.List" %>
<%@ page import="com.google.appengine.api.users.User" %>
<%@ page import="com.google.appengine.api.users.UserService" %>
<%@ page import="com.google.appengine.api.users.UserServiceFactory" %>
<%@ page import="com.google.appengine.api.datastore.DatastoreServiceFactory" %>
<%@ page import="com.google.appengine.api.datastore.DatastoreService" %>
<%@ page import="com.google.appengine.api.datastore.Query" %>
<%@ page import="com.google.appengine.api.datastore.Entity" %>
<%@ page import="com.google.appengine.api.datastore.FetchOptions" %>
<%@ page import="com.google.appengine.api.datastore.Key" %>
<%@ page import="com.google.appengine.api.datastore.KeyFactory" %>
<%@ taglib prefix="fn" uri="http://java.sun.com/jsp/jstl/functions" %>

<html>
  <head>
    <link type="text/css" rel="stylesheet" href="/stylesheets/main.css" />
  </head>

  <body>

<%
    String guestbookName = request.getParameter("guestbookName");
    if (guestbookName == null) {
        guestbookName = "default";
    }
    pageContext.setAttribute("guestbookName", guestbookName);
    UserService userService = UserServiceFactory.getUserService();
    User user = userService.getCurrentUser();
    if (user != null) {
      pageContext.setAttribute("user", user);
%>
<p>Hello, ${fn:escapeXml(user.nickname)}! (You can
<a href="<%= userService.createLogoutURL(request.getRequestURI()) %>">sign out</a>.)</p>
<%
    } else {
%>
<p>Hello!
<a href="<%= userService.createLoginURL(request.getRequestURI()) %>">Sign in</a>
to include your name with greetings you post.</p>
<%
    }
%>

<%
    DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
    Key guestbookKey = KeyFactory.createKey("Guestbook", guestbookName);
    // Run an ancestor query to ensure we see the most up-to-date
    // view of the Greetings belonging to the selected Guestbook.
    Query query = new Query("Greeting", guestbookKey).addSort("date", Query.SortDirection.DESCENDING);
    List<Entity> greetings = datastore.prepare(query).asList(FetchOptions.Builder.withLimit(5));
    if (greetings.isEmpty()) {
        %>
        <p>Guestbook '${fn:escapeXml(guestbookName)}' has no messages.</p>
        <%
    } else {
        %>
        <p>Messages in Guestbook '${fn:escapeXml(guestbookName)}'.</p>
        <%
        for (Entity greeting : greetings) {
            pageContext.setAttribute("greeting_content",
                                     greeting.getProperty("content"));
            if (greeting.getProperty("user") == null) {
                %>
                <p>An anonymous person wrote:</p>
                <%
            } else {
                pageContext.setAttribute("greeting_user",
                                         greeting.getProperty("user"));
                %>
                <p><b>${fn:escapeXml(greeting_user.nickname)}</b> wrote:</p>
                <%
            }
            %>
            <blockquote>${fn:escapeXml(greeting_content)}</blockquote>
            <%
        }
    }
%>

    <form action="/sign" method="post">
      <div><textarea name="content" rows="3" cols="60"></textarea></div>
      <div><input type="submit" value="Post Greeting" /></div>
      <input type="hidden" name="guestbookName" value="${fn:escapeXml(guestbookName)}"/>
    </form>

  </body>
</html>

Warning! Whenever you display user-supplied text in HTML, you must escape the string using the fn:escapeXml JSTL function, or a similar escaping mechanism. If you do not correctly and consistently escape user-supplied data, the user could supply a malicious script as text input, causing harm to later visitors.

Retrieving the Stored Greetings

The low-level Java API provides a Query class for constructing queries and a PreparedQuery class for fetching and returning the entities that match the query from the Datastore. The code that fetches the data is here:

    Query query = new Query("Greeting", guestbookKey).addSort("date", Query.SortDirection.DESCENDING);
    List<Entity> greetings = datastore.prepare(query).asList(FetchOptions.Builder.withLimit(5));

This code creates a new query on the Greeting entity, and sets the guestbookKey as the required parent entity for all entities that will be returned. We also sort on the date property, returning the newest Greeting first.

After you construct the query, it is prepared and returned as a list of Entity objects. For a description of the Query and PreparedQuery interfaces, see the Datastore reference.

A Word About Datastore Indexes

Every query in the App Engine Datastore is computed from one or more indexes. Indexes are tables that map ordered property values to entity keys. This is how App Engine is able to serve results quickly regardless of the size of your application’s Datastore. Many queries can be computed from the builtin indexes, but the Datastore requires you to specify a custom index for some, more complex, queries. Without a custom index, the Datastore can’t execute the query efficiently.

Our guest book example above, which filters results by ancestor and orders by date, uses an ancestor query and a sort order. This query requires a custom index to be specified in your application’s datastore-indexes.xml file. When you run your application in the SDK, it will automatically add an entry to this file. When you upload your application, the custom index definition will be automatically uploaded, too. The entry for this query will look like:

<?xml version="1.0" encoding="utf-8"?>
<datastore-indexes
  autoGenerate="true">
    <datastore-index kind="Greeting" ancestor="true">
        <property name="date" direction="desc" />
    </datastore-index>
</datastore-indexes>

You can read all about Datastore indexes on the Datastore Indexes page.

Clearing the Datastore

The development web server uses a local version of the Datastore for testing your application, using local files. The data persists as long as the temporary files exist, and the web server does not reset these files unless you ask it to do so.

The file is named local_db.bin, and it is created in your application’s WAR directory, in the WEB-INF/appengine-generated/ directory. To clear the Datastore, delete this file.

Using Static Files

There are many cases where you want to serve static files directly to the web browser. Images, CSS stylesheets, JavaScript code, movies and Flash animations are all typically served directly to the browser. For efficiency, App Engine serves static files from separate servers than those that invoke servlets.

By default, App Engine makes all files in the WAR available as static files except JSPs and files in WEB-INF/. Any request for a URL whose path matches a static file serves the file directly to the browser—even if the path also matches a servlet or filter mapping. You can configure which files App Engine treats as static files using the appengine-web.xml file.

Let’s spruce up our guest book’s appearance with a CSS stylesheet. For this example, we will not change the configuration for static files. See App Configuration for more information on configuring static files and resource files.

A Simple Stylesheet

In the directory war/, create a directory named stylesheets/. In this directory, create a file named main.css with the following contents:

body {
    font-family: Verdana, Helvetica, sans-serif;
    background-color: #FFFFCC;
}

Edit war/guestbook.jsp and insert the following lines just after the <html> line at the top:

<html>
  <head>
    <link type="text/css" rel="stylesheet" href="/stylesheets/main.css" />
  </head>

  <body>
    ...
  </body>
</html>

Visit http://localhost:8888/. The new version uses the stylesheet.

Uploading Your Application

You create and manage applications in App Engine using the Administration Console.

Once you have registered an application ID for your application, you upload it to App Engine using either the Eclipse plugin, or a command-line tool in the SDK.

Note: Once you register an application ID, you can delete it, but you can’t re-register that same application ID after it has been deleted. You can skip these next steps if you don’t want to register an ID at this time.

Note : If you have an App Engine Premier account, you can specify that your new application should reside in the European Union rather than the United States. For developers that do not have a Premier account, you will need to enable billing for applications that should reside in the European Union.

Hosting applications in the European Union is especially useful if your users are closer to Europe than to the United States. There is less network latency and the End User Content will be stored at rest in the European Union. You must specify this location by clicking the “Edit” link in the “Location Options” section when you register the application; you cannot change it later.

Registering the Application

You create and manage App Engine web applications from the App Engine Administration Console, at the following URL:

https://appengine.google.com/

Sign in to App Engine using your Google account. If you do not have a Google account, you can create a Google account with an email address and password.

To create a new application, click the “Create an Application” button. Follow the instructions to register an application ID, a name unique to this application.

Edit the appengine-web.xml file, then change the value of the <application> element to be your registered application ID.

For this tutorial, you should probably elect to use the free appspot.com domain name, and so the full URL for the application will be http://your_app_id.appspot.com/. You can also purchase a top-level domain name for your app, or use one that you have already registered.

For Authentication Options (Advanced), the default option, “Open to all Google Accounts users“, is the simplest choice for this tutorial. If you choose “Restricted to the following Google Apps domain”, then your domain administrator must add your new app as a service on that domain. If you choose the Google Apps domain authentication option, then failure to add your app to your Google Apps domain will result in an HTTP 500 where the stack trace shows the error “Unexpected exception from servlet: java.lang.IllegalArgumentException: The requested URL was not allowed: /guestbook.jsp”. If you see this error, add the app to your domain. See Configuring Google Apps to Authenticate on Appspot for instructions.

If you have an App Engine Premier account, you can specify that your new application should reside in the European Union rather than the United States. This is especially useful if your application’s users are closer to Europe than to the United States. There is less network latency and the End User Content will be stored at rest in the European Union. You must specify this location when you register the application; you cannot change it later. Click the Edit link in the Location Options section; select a location option, either United States or European Union.

Uploading the Application

You can upload your application using Eclipse, or using a command at the command prompt.

Currently, you can only upload applications with a maximum size of 32 megabytes.

Uploading From Eclipse

You can upload your application code and files from within Eclipse using the Google Plugin.

To upload your application from Eclipse, click on the Google button  in the Eclipse toolbar, then select “Deploy to App Engine.”

If prompted, follow the instructions to provide the Application ID from the App Engine console that you would like to use for this app, your Google account username (your email address), and your password. Then click the Deploy button. Eclipse will then automatically upload the contents of the war/ directory.

Uploading Using the Command Prompt

You can upload your application code and files using a command included in the SDK named appcfg.cmd (Windows) or appcfg.sh (Mac OS X, Linux).

AppCfg is a multi-purpose tool for interacting with your app on App Engine. The command takes the name of an action, the path to your app’s war/ directory, and other options. To upload the app code and files to App Engine, you use the update action.

To upload the app, using Windows:

..appengine-java-sdkbinappcfg.cmd update war

To upload the app, using Mac OS X or Linux:

../appengine-java-sdk/bin/appcfg.sh update war

Enter your Google username and password at the prompts.

Checking Your Application State

After your application is uploaded, its Datastore Indexes will be automatically generated. This operation may take some time, and any visitors to your site will receive aDatastoreNeedIndexException until the indexes have been built. You can monitor the progress of the operation by visiting the App Engine console, selecting your application, and then selecting the Datastore Indexes link.

Accessing Your Application

You can now see your application running on App Engine. If you set up a free appspot.com domain name, the URL for your website begins with your application ID:

http://your_app_id.appspot.com/

Congratulations!

You have completed this tutorial. For more information on the subjects covered here, see the rest of the App Engine documentation.


GAE: Introducción al App Engine Java

Con App Engine , podemos construir aplicaciones web utilizando tecnologia Java estandar y ejecutarlas en las infraestructuras escalables de Google. El entorno Java proporciona una máquina virtual Java 6 , un interfaz de Servlets de Java y soporte para interfaces estandares para acceder a los datastores y servicios escalables de Google, tales como JDO, JPA, JavaMail y JCache. Al seguir estandares hace fácil el desarrollo de las aplicaciones y mantienes la posibilidad de que en el futuro puedes instalartelo en tu propio entorno de ejecución de aplicaciones web.

El Google Plugin for Eclipse nos ofrece la posibilidad de crear proyectos en formato wizard y varias opciones de configuración de depuración de tus proyectos en GAE. Con el App Engine de Java se hace especialmente fácil programar y desplegar aplicaciones web utilizando el Google Web Toolkit (GWT). El plugin de eclipse viene con los SDKs de App Engine y de GWT.

Hay plugins de terceros también disponibles para otros IDEs de Java.For NetBeans, see NetBeans support for Google App Engine. For IntelliJ, see Google App Engine Integration for IntelliJ. (These links take you to third-party websites.)

Si no lo has hecho aún , échale un vistazo a la the Java Getting Started Guide para tener una introducción interactiva de lo que es programar aplicaciones web con tecnologías Java y GAE

The Java Runtime Environment

App Engine runs Java applications using the Java 6 virtual machine (JVM). The App Engine SDK supports Java 5 and later, and the Java 6 JVM can use classes compiled with any version of the Java compiler up to Java 6.

App Engine uses the Java Servlet standard for web applications. You provide your app’s servlet classes, JavaServer Pages (JSPs), static files and data files, along with the deployment descriptor (the web.xml file) and other configuration files, in a standard WAR directory structure. App Engine serves requests by invoking servlets according to the deployment descriptor.

The JVM runs in a secured “sandbox” environment to isolate your application for service and security. The sandbox ensures that apps can only perform actions that do not interfere with the performance and scalability of other apps. For instance, an app cannot spawn threads in some ways, write data to the local file system or make arbitrary network connections. An app also cannot use JNI or other native code. The JVM can execute any Java bytecode that operates within the sandbox restrictions.

See Servlet Environment for more information.

The Datastore, the Services and the Standard Interfaces

App Engine provides scalable services that apps can use to store persistent data, access resources over the network, and perform other tasks like manipulating image data. You have the choice between two different data storage options differentiated by their availability and consistency guarantees. Where possible, the Java interfaces to these services conform to established standard APIs to allow for porting apps to and from App Engine. Each service also provides a complete low-level interface for implementing new interface adapters, or for direct access.

Apps can use the App Engine datastore for reliable, scalable persistent storage of data. The datastore supports two standard Java interfaces: Java Data Objects (JDO) 2.3 and Java Persistence API (JPA) 1.0. These interfaces are implemented using DataNucleus Access Platform, the open source implementation of these standards.

The App Engine Memcache provides fast, transient distributed storage for caching the results of datastore queries and calculations. The Java interface implements JCache (JSR 107).

Apps use the URL Fetch service to access resources over the web, and to communicate with other hosts using the HTTP and HTTPS protocols. Java apps can simply use java.net.URLConnection and related classes from the Java standard library to access this service.

An app can use the Mail service to send email messages on behalf of the application’s administrators, or on behalf of the currently signed-in user. Java apps use the JavaMailinterface for sending email messages.

The Images service lets applications transform and manipulate image data in several formats, including cropping, rotating, resizing, and photo color enhancement. The service can handle CPU-intensive image processing tasks, leaving more resources available for the application server to handle web requests. (You can also use any JVM-based image processing software on the application server, provided it operates within the sandbox restrictions.)

An application can use Google Accounts for user authentication. Google Accounts handles user account creation and sign-in, and a user that already has a Google account (such as a GMail account) can use that account with your app. An app can detect when the current user is signed in, and can access the user’s email address. Java applications can use security constraints in the deployment descriptor to control access via Google Accounts, and can detect whether the user is signed in and get the email address using the getUserPrincipal() method on the servlet request object. An app can use the low-level Google Accounts API to generate sign-in and sign-out URLs, and to get a user data object suitable for storage in the datastore.

Scheduled Tasks

An application can configure scheduled tasks that will call URLs of the application at specified intervals. For more on this, see Cron Jobs.

Java Tools

The App Engine Java SDK includes tools for testing your application, uploading your application files, and downloading log data. The SDK also includes a component for Apache Ant to simplify tasks common to App Engine projects. The Google Plugin for Eclipse adds features to the Eclipse IDE for App Engine development, testing and deployment, and includes the complete App Engine SDK. The Eclipse plugin also makes it easy to develop Google Web Toolkit applications and run them on App Engine.

The development server runs your application on your local computer for development and testing. The server simulates the App Engine datastore, services and sandbox restrictions. The development server can also generate configuration for datastore indexes based on the queries the app performs during testing.

A multipurpose tool called AppCfg handles all command-line interaction with your application running on App Engine. AppCfg can upload your application to App Engine, or just update the datastore index configuration so you can build new indexes before updating the code. It can also download the app’s log data, so you can analyze your app’s performance using your own tools.

Opera Mini: guía para maquetadores

[Fuente: http://dev.opera.com/articles/view/opera-mini-web-content-authoring-guidelines/]

Indice

Introducción

Este documento tiene dos propositos:

  • Explicar el estado actual de navegación por internet con móviles
  • Detallar las cosas que hay que tener en cuenta cuando se diseñan páginas con Opera Mini

Mobile browsing in a nutshell

Mobile browsing is generally very resource-limited. There are three broad approaches on how to deal with Web content in such environments. Each one has its tradeoffs and advantages, and the potential for supporting Web standards (and working with websites not designed for mobile use) is greater in some cases than in others.

First, there are browsers that run the same engine as desktop browsers, such as Opera Mobile and Safari on the iPhone. Since the same engine is used, websites are supported as well as they are on desktop browsers. Most devices do not have enough CPU power or memory to run these browsers and so they tend to only be found on high-end phones.

Second, there are browsers designed specifically for mobile: these generally have limited support for Web standards. Some of these only support WAP browsing (such as the OpenWave browser), some support other standards like cHTML or XHTML MP (such as the Japanese NTT DoCoMo iMode browsers), and some support a limited subset of general Web standards (such as Netfront, Pocket IE, and Blazer).

Third, there are browsers that use a proxy system to deliver content to the phone. In this setup, there is a client on the device that connects to a proprietary server, which then connects to the wider web for page requests, does some server-side processing and sends content back the client. Opera Mini falls into this category: when the user navigates to a new page, the server fetches the page and transforms it into a lightweight compressed binary markup language called OBML. This decreases bandwidth usage by up to 90% and saves CPU usage on mobile devices, and as such makes browsing on resource-restricted devices much more viable. It also has speed benefits, since the majority of data transfer occurs between the Mini servers and website servers, both of which are on high-speed connections.

Developing for the mobile web

When developing for the mobile platform, you can choose to either build a single site designed to work well in all browsers with various browser– and device–specific tweaks, or you can build two entirely separate sites, one optimised for desktop browsers and the other for mobile browsers.

Opera strongly recommends building one site for all devices. We believe in one Web that works across platforms and devices, and this is why we work hard to create browsers that support Web standards as well as possible and use the same rendering engine for all of our products. Developing one site has the advantage of reducing development, QA, and maintenance time, and with the growth of standards-supporting browsers, it is an achievable goal.

That said, developing a separate site can sometimes make sense. If you already have a site that is designed to be highly interactive and you now want it accessible on mobile devices, it may be easier to develop a cut-down version than to retrofit the changes to the existing design. The decision should ultimately be made by weighing up the possibility of an improved user experience against the extra cost and maintenance of a separate site.

If you do create a separate site, be aware that serving different content based on the browser is not foolproof. Make sure you include links from the mobile version of the site to the full version, so that if you misidentify the browser or if the user is using a mobile browser that can handle the full site, the user is not blocked from your content.

Design considerations

Mobile devices are resource-constrained in various ways that desktop browsers are not; the most important of these are listed below:

  • Limited control. Most mobile devices don’t have pointing devices like a computer mouse. Most don’t have a traditional keyboard either, and text input might take a lot of effort. To help mitigate these problems, you should make as many options selectable from drop-down lists or radio buttons as possible, and consider prefilling fields with likely choices.
  • Reduced screen size: Devices have wildly varying resolutions. 480×320 would be a large screen for a mobile device, and many have screens as small as or smaller than 240×320. As a result, complicated layouts that require large resolutions should be avoided on mobile devices, since they will only make the user have to scroll. This can be done by keeping a design deliberately simple, sending different pages to mobile devices, or using CSS media queries to change layouts dynamically.
  • Limited memory and bandwidth: Mobile devices will obviously have less memory available than desktop computers, so you need to think carefully about the length of your pages and how many images you use on them. Some mobile browsers have the option of turning images off, so you should make sure that your images have text alternatives available.
  • Limited web standards support: As mentioned above, some devices have support for a limited subset of web standards, so make sure your pages are designed with graceful degradation in mind, and/or do feature detection to work out if browsers support advanced features before serving them, and then provide alternatives to those that don’t. Be careful to use JavaScript in such a way that if scripting support is turned off or some features are not present, your pages’ main functionality will still be available. We’ll discuss these concepts more later on — see HTML and CSS support below.

To design and create your site such that it will work to an acceptable level in the greatest number of browsers without individually testing each and every one, ensure that the HTML you produce is well-structured and in a logical order. This will ensure that the page will still be intelligible and legible on devices that support minimal styling and/or JavaScript. You can simulate this by disabling style/script in your browser (Opera and Firefox both support this). Putting things in a logical order is important because some browsers (Mini included) will sometimes collapse complex layouts so they are displayed on the screen in a single column, thus preventing horizontal scrolling. If your content is written in a sensible order, then this feature will work much better. For more information, please see the section on rendering modes below.

HTML and CSS support

Opera Mini is built on the same core rendering engine as Opera Desktop, and so it has excellent support for Web standards, including HTML, CSS, XSLT, XPath, and SVG. However, there are some differences.

Basic HTML and CSS

Mini supports borders and backgrounds, including asymmetric styles and sizes. Dotted and dashed borders are however rendered as solid for bandwidth and memory reasons. Tables are supported, but bear in mind that even three-column tables might force the user to scroll horizontally in order to see all the data. Using tables for layout should be avoided at all costs.

Mini does not support the line-height CSS property at present, since testing showed that it generally meant less text fitted on any individual page, requiring more scrolling from the user. As a result, using line-height to position things vertically will not work, like in the example below:

<p style="height: 52px; line-height: 52px;">Text</p>

This would result in the text being displayed at the top of the box rather than in the middle. While Mini supports frames, we strongly recommend you do not use them; there are better ways to achieve the same effect. Mini also supports iframes, though with some limitations. Unlike on desktop browsers, Mini has only one scrollable area – the <body>element — and iframes will have their contents cut off if they are not large enough to display them. Other elements that use the CSS to create scrollable sections within the page — for example overflow-y — will also be cut off in the same way.

Rendering modes

Mini has two different rendering modes, desktop and mobile, and the user can freely switch between them:

  • Desktop rendering renders pages in much the same way as a desktop browser, with the exception that it will try to force columns of text to not exceed the width of the screen, to reduce the need for horizontal scrolling.If the web page is wider than the device screen, a virtual mouse pointer is shown that the user can use to scroll around. If the page’s width is greater than twice the width of the device’s screen, then overview mode is triggered, which shows a zoomed-out view of the page that fits the screen horizontally and scrools vertically. The user can then choose an area to zoom in and read at full size.
  • In mobile view, Opera reformats pages so that they fit into a single column. This keeps the elements of the original page design (such as colours and images) but linearises their display in an attempt to eliminate horizontal scrolling.If mobile view is turned on and the page provides a handheld stylesheet, as shown below, then the layout will not be reflowed or linearised, and that stylesheet will be used instead:
    <link rel="stylesheet" type="text/css" media="handheld" href="handheld.css">

Image formats, animation, and video

Opera Mini supports all the image formats supported by its desktop cousin, including PNG, JPEG, GIF, SVG, BMP, ICO, TGA, and WBMP. However, the Mini servers recompress images at a lower quality setting to save bandwidth, as well as sometimes resizing them so that they fit better on the user’s screen. The user can change this behaviour and opt to have higher-quality images if they want, but this is not the default.

Because of this compression, and for accessibility reasons, we strongly recommend that you do not use images to display text. If you must do so (e.g. because you are using image replacement for a header font), then include the image using CSS and optimise the image’s size and clarity for Mini’s low quality mode. You can also use different stylesheets for screen and handheld browsers and include the image only on the former, though this only works in Mobile view. Regardless, always make sure the original text is available for screen readers to access, whether with alt text or replaced content.

Animation of any kind (including animated GIFs, animated SVG, or the <blink> or <marquee> elements) is not supported in Opera Mini because the server pre-renders the page and merely sends the client a snapshot of it.

If Mini encounters a link to an RTSP stream, then it is handed to the media player on the client device to deal with. This does not take place over the Mini proxy but via a direct connection, and so may not be possible on some devices.

Typography

Mini uses the fonts present on the device it runs on, and the native font on most devices is a sans-serif one. Most devices will only have a few font sizes, so Mini tries to match what web pages request with what is available on the device. This is not always possible, and so designs that expect pixel-perfect layout will probably not work. Designs should be flexible enough to handle font sizes other than what they expect.

When it comes to font families, Opera Mini will only use one family of font per page, and setting font-family will have no effect, except in the case of monospace fonts where it will simulate monospace. Mini also includes a small bitmap font that covers the ISO-8859-1 character range and is often the preferred choice on devices with low resolution and large system fonts.

Mini supports bold fonts, and the text-decoration property values underlineoverline, and strike, but notitalic. It also supports the <sub> and <sup> elements.

Downloads

Downloads are either launched in the native device browser, or performed from within Mini itself, depending on the functionality of the Java environment on the device. Regardless of which application on the device actually opens the download, it will be proxied through the same Mini transcoder server that the Mini client is connected to. This allows downloads that require state — such as cookies — to be successful.

Metadata

Mini supports basic Web page metadata such as document title and favicons, both of which are shown at the top of the browser. It is recommended to use the <link> element with a rel value of shortcut icon to locate the relevant icon for each page, like so:

<link rel="shortcut icon" type="image/png" href="icon.png">

Mini also supports the alternate rel value to point to feeds that can be displayed and subscribed to:

<link rel="alternate" type="application/rss+xml" href="/feed.rss">

If such feeds are present, a webpage will display an RSS icon in the top left hand corner of the Mini display.

JavaScript support

Mini supports JavaScript as well as Opera Desktop, but instead of running on the client, it runs on the server. A useful way to think about it is that the Mini servers are running web browsers in virtual windows that render their output to the phone. All the phone does is act as a display mechanism and a set of input controls for the server.

The server doesn’t render to the phone constantly; instead, it takes a snapshot of the page after it has been loaded, pauses all running scripts, and sends that to the phone. This means that while Mini supports JavaScript just as well as Opera Desktop, interactivity is somewhat more limited. When the user clicks a link or a button, then the Mini client sends that information back to the server, where the server performs the associated action (such as loading a new page, executing some JavaScript, etc.).

For authors, the upshot of all this is that once a page has been rendered by the server, it won’t change until the user does something on that and there is no way for scripts to run in the background. The user must do something to make Mini talk to the server in order for JavaScript to be unpaused. As a result, you cannot expect things like JavaScript animations or timed Ajax updates to work in the background as they would on a desktop browser.

JavaScript running on the Mini server will only run for a couple of seconds before pausing, for resource constraint reasons. This applies to JavaScript run due to an event firing e.g. onload, as well as code run because of a user action.

Supported events

The load and unload events are triggered the same as they are in desktop browsers. All elements with mousedown,mouseup or click event listeners, as well as hyperlinks, will be turned into selectable areas on the Mini client. When clicked by the user, those elements will receive the mouseovermousedownmouseup and click events, in that order, as you would expect on a desktop browser. Note that Mini does not send the mouseout event and that mouseover is only triggered if a click happens.

To reduce the number of server roundtrips, the Mini client only sends filled-in form data to the server when the submit button is clicked, or when an <input> element has a change event listener. This means that form-related events are all fired at once when the form is submitted or when a change event is triggered. In that case Mini will go over all the visible input controls on the page and fire the appropriate focus, click and change events on them, in that order. The change event immediately causes a roundtrip so that for example, data validation can happen straight away, rather than waiting until the form is submitted.

On forms that have one only visible text input control, Mini will submit the form to the server as soon as the contents of that control are changed by the user. Mini doesn’t support any mouse or keypress events not mentioned above, nor resize or scroll events: this is to save bandwidth and prevent constant server roundtrips.

Opera Mini does not support popup windows. If you use window.open(), you will find that the opened window replaces the previously open window. In addition, calls to window.close() will be ignored — the user will have to use the ‘back’ button manually.

Ajax support

As a result of scripts being kept paused on the server while there is no user interaction, “Ajax” techniques may not work as expected. Code registered with setTimeout() and setInterval() may be run before the page is sent, but it is unlikely that it will be called more than once because of script pausing. XMLHttpRequest is supported just as it is on desktop browsers, but triggering such requests from timeouts and intervals will not cause dynamic page updates, since that code will not be run in the background.

Things affected by this include JavaScript animations that use setInterval to move or resize elements (with the effect that pages may be sent to the client mid-move or mid-resize), IRC-client chat clients, clocks, background polling for new mail or notifications, and dynamic form autocomplete suggestions.

operamini object

Mini provides a special JavaScript object — operamini — that allows access to various special features of Mini and the device it is running on. This object is detailed in the rest of this section and in the following section.

If you want to detect operamini, you should do so with a check on the operamini object like so:

Object.prototype.toString.call(window.operamini) === "[object OperaMini]"

This way of doing things is better than a simple variable check, as an object’s internal Class can’t be tampered with — in this case there is no way to set an arbitrary window.operamini variable to pass false results, as there would be with this:

 if (operamini) {
    alert("Running on Opera Mini");
}

operamini.page object

Setting operamini.page.maxAge will change the maximum amount of time a page is kept in Mini’s cache on the client side, measured in minutes. Setting it to 0 will mean that a page is never cached. Values above 65535 will be cut down to 65535 (around 45 days).

Setting operamini.page.samePage to true will tell the server that the page should not be zoomed out when the page is loaded, but should remain zoomed in at the location it was on the previous page. It is intended to be used in situations where you navigate to a new page but want to keep the same focus area.

operamini.sms object

Opera Mini has the ability to send SMS messages on devices that support them. You can use this functionality by setting the number and body properties of the operamini.sms object to the number you want to send an SMS to, and the contents of the SMS, respectively. Devices will prompt the user before sending an SMS and some may allow editing of the message and/or number before sending. Example usage of this functionality:

<script>
function send() {
  operamini.sms.number = document.getElementById("number").textContent;
  operamini.sms.body = document.getElementById("body").textContent;
}
</script>

<p>Number: <input type="text" id="number" />
<p>Body: <input type="text" id="body" />
<p><button onclick="send()">Send SMS</button>

You can detect whether you can send SMS messages before trying to by switching your code based on the value ofoperamini.features.sms.

Customising content for Mini

There are various ways to customise the content you provide — either on the web server or via JavaScript. These are detailed below.

User-Agent sniffing

A browser can be identified by checking the user agent string, which is sent as the User-Agent header in any HTTP requests it makes. Mini’s user agent string comes in this form:

Opera/9.80 ($PLATFORM; Opera Mini/$CLIENT_VERSION/$SERVER_BUILD; U; $CLIENT_LANG) Presto/$PRESTO_VER Version/$OPERA_VER

for example:

Opera/9.80 (J2ME/MIDP; Opera Mini/6.1.25403/25.842; U; en) Presto/2.5.25 Version/10.54

Phone manufacturers and mobile carriers often customise browser user-agent strings, sometimes on a site-specific basis, so testing the user agent string is discouraged. Instead, you should detect individual features present on the device you are sending to. This approach is more flexible and future-proof, and means that if the Opera Mini servers are upgraded (which may lead to an adjusted UA string), then you can continue to support Opera Mini with no extra effort on your part.

If you absolutely need to serve content based on the user-agent string, then you should test based on the presence of the substring Opera Mini.

Feature strings

There are two main ways to detect what functionality is available for use on the phone when using Mini: using HTTP request headers, or in-page using JavaScript. You can find detailed information on what HTTP headers Opera Mini has available for use in the article Opera Mini request headers.

Opera Mini provides various feature strings. These can be accessed through the operamini.features object in JavaScript:

if (operamini.features.sms) {
  // send SMS message using operamini.sms object (see JavaScript section)
}

Alternatively, you can test this via the X-OperaMini-Features HTTP request header, which sends results in the form of a comma-separated list, like so:

X-OperaMini-Features: advanced, folding, secure

The feature strings available can be seen in X-OperaMini-Features.

Detecting the device’s native user-agent

Mini provides access to the device’s native user-agent string, or a best guess at it, through the operamini.remoteuaJavaScript object or the X-OperaMini-Phone-UA HTTP header. Devices that come with Mini preinstalled have this value hardcoded when Mini is embedded on the device. When Mini is downloaded and installed by a user, the User-Agent string of the client downloading the .jad file will be used instead. If neither of these routes prove fruitful, Mini will look at the Java environment on the device and use heuristics to pick a UA string that has a good chance of being right. If all else fails, this value will be unknown.

Detecting the device’s manufacturer and model

Opera Mini provides access to the device’s manufacturer and model details through the operamini.phone JavaScript object or the X-OperaMini-Phone HTTP header, provided that the device makes this information available to Opera Mini.

The format of the string provided in either case is [manufacturer # model], so for example Nokia # E75-1 orSonyEricsson # K750i. If neither can be determined, the value of that field will be unknown.

Detecting the real source of a request

Most HTTP proxy servers, including Mini, make use of the X-Forwarded-For HTTP header. Each proxy server appends the IP address it is proxying the request for to the end of the string, in a comma-separated list. The header then looks like this:

X-Forwarded-For: client, proxy1, proxy2

This means that a proxy (the Mini proxy) forwarded the request for proxy2, which forwarded it for proxy1, which forwarded it for clientclient is the most reliable source of information about the origin of the request, and is suitable for geolocation.

Note that to work around bugs in some proxies, Opera Mini sometimes sends this header as lowercased (x-forwarded-for), so you should check for it case-insensitively. The first value in the list (e.g. client, from above) is also accessible via JavaScript using operamini.remote.

Detecting the user’s language

Opera Mini, along with most other browsers, sends the Accept-Language HTTP header with its page requests.Accept-Language specifies what language(s) the browser would prefer the HTTP response to be in using standard ISO-639 language tags in a comma-delimited list. Each language is given a “quality” value that indicates the user’s ability in that language. The quality can range from 0 to 1, with 1 being most preferable and 0 least.

In practice, the contents of this header will be determined by the current language of the device making the request (this is the case on both desktop and mobile browsers). If you have content in multiple languages, you should also provide an in-page way to switch between them because Accept-Language strings are not always accurate. Other circumstances may also arise, for example you may have a native German speaker using someone else’s phone, the language of which is set to Afrikaans, or another language.

The example given below means “I’d prefer to be sent old Norwegian (bokmal) or new Norwegian, but I am also pretty good at English; I’ll try French if you’ve not got the other languages available, but I’m not very good at that.” In practice,Accept-Language strings are usually much simpler and generally only contain one or two language tags.

Format:

Accept-Language: <language tag>;q=<quality value>, ...

Example:

Accept-Language: no-bok, no-nyn, en;q=0.8, fr;q=0.4

Media types

CSS and HTML have a mechanism for specifying that certain stylesheets or portions of stylesheets should be used on certain display devices: “media types”. Here are some examples:

<link rel="stylesheet" media="screen" type="text/css" href="main.css" />
<link rel="stylesheet" media="print" type="text/css" href="print.css" />
<link rel="stylesheet" media="handheld" type="text/css" href="mobile.css" />

Opera Mini uses the screen media type by default, since that it closest to its capabilities. The handheld type will only be used if the user has explicitly switched to mobile view, in which case regular small screen rendering will be overridden by the handheld stylesheet.

Media queries

Media queries are a CSS3 feature that allow page styling to vary with the characteristics of the device a document is being displayed on, for example screen width, aspect ratio, resolution, and so on. An example of a stylesheet using media queries is given below:

/* Two-column layout for most screens */

#navigation { float: left; width: 40%; }
#content { float: right; width: 60%; }
img { margin: 10px; }

/* Media query 1: Disable some features when the page is less than or equal to 480px */

@media all and (max-width: 480px) {

  /* Don"t float, make the page linear, and center images */

  #navigation, #content { float: none; width: 100%; }
  img { margin: 10px auto; display: block; }

}

/* Media query 2: Disable some things when the page is less than or equal to 240px */

@media all and (max-width: 240px) {

  /* Remove purely decorative images */
  img.decorative { display: none; }

}

In this example, we have a basic two-column layout with a navigation column and a content column, and images have a 10px margin. When the browser’s viewport is less than or equal to 480px, we stop it being two-column and instead collapse it so that the content and navigation sections follow each other in a single column. We also center images rather than allowing text to flow around them. Then, at viewport widths less than or equal to 240px, we stop displaying all images marked with a class of “decorative”.

Browser support for media queries is getting better but is not yet universal. On the desktop, Opera 7+, Safari 3+, Firefox 3.5+ and Internet Explorer 9+ support them. On mobile devices, Opera and WebKit-based browsers do, though most others do not. One way to tackle this is to use a combination of media types and media queries, with one stylesheet for screen users, small screen rules inside that stylesheet that override the ones in the broad stylesheet, and then a separate handheld stylesheet for less advanced browsers:

<style type="text/css" media="screen,projection">
/* rules for desktop and other devices with lots of space */
@media only all and (max-width: 480px) {
/* override rules for smaller-screened devices */
}
</style>

<style type="text/css" media="handheld">
/* rules for mobile devices that don't support media queries */
</style>

With this approach, desktop users will get the full experience, more advanced mobile browsers will parse the media queries and resize content according to device sizing, and less advanced mobile browsers that do not understand media queries (Pocket IE, Netfront, OpenWave, and early versions of Opera Mini) can receive a stripped-down design.

Debugging Websites for Opera Mini

In this section we will look at the features available for debugging on Opera Mini.

Tools

There are a number of useful tools available for debugging sites running on Opera Mini.

Resizable Emulator

Opera Mini wraps a website’s text and shrinks images to fit each device’s screen. Of course, this means the website’s look will depend on the device’s screen size, which calls for authors to test their sites with more than one device, or better, a resizable emulator.

A good choice for trying websites with Opera Mini is the MicroEmulator. Inside the downloadable package, there is a “devices” subdirectory containing a resizable skin. To use it, start the emulator, select options > select device > add, and open the file microemu-device-resizable.jar before loading the Opera Mini MIDlet (the easiest way to get this is from the Opera Mini mobile page — go here, and select “Other download options” for a list of different MIDlets). If you get stuck, there is a useful guide available: Using Microemulator to Run Opera Mini.

It is also recommended to test the website with different font sizes (which can be changed in Opera Mini’s settings page). Be aware that font size on physical devices may differ from the emulator, since the actual font size depends on the underlying platform.

Opera Dragonfly

Since the Opera Mini servers use the same rendering engine as Opera Desktop, a lot of what is going on can be inspected simply by opening a website in Opera Desktop and starting the Opera Dragonfly developer tool (Tools > Advanced > Developer Tools).

View page source

It is possible to get a dump of the current page source by entering server:source into Opera Mini’s address bar. This shows the current DOM tree from the Opera Mini server. Since reading the source from a cell phone display or even an enlarged microemulator isn’t very pleasant, there is an even better way of obtaining a page’s source code — posting it to a webserver by entering server:source?post=http://your.server.com/script into Opera Mini’s address bar, where the script file that is passed the post data will handle the request, e.g. by storing the source in a database, or saving it to a text file. The posted data includes the fields urlhost and html.

Solving common problems

One of the more common layout issues that occurs on Mini is when we want certain content not to be wrapped, for example horizontal menu bars. This is quite easy to solve: Opera Mini won’t wrap lines between floated items as long as they fit in their container’s width. Non-breaking spaces are honored as well in case normal text isn’t wrapped.

Another common issue is that many websites depend on a minimum window width to look good. If a site layout looks too narrow in Opera Mini, the problem usually also affects desktop browsers with a small window width. Giving the body a minimum width (min-width) is thus often the solution for both Opera Mini and desktop browsers running on lower resolution displays.

This article is licensed under a Creative Commons Attribution 3.0 Unported license.

Comments

The forum archive of this article is still available on My Opera.

Android, iOS, tiempos de respuestas y por qué nada es gratis en sistemas informáticos

[Fuente: http://gallir.wordpress.com/2011/12/07/android-ios-tiempos-de-respuestas-y-por-que-nada-es-gratis-en-sistemas-informaticos/]

Hace unas pocas horas escribí esta respuesta sobre Por qué iOS es más fluido que Android (con buen criterio, eliminaron la entrada). Obviamente, por cuestiones de longitud y la “respuesta rápida” que requiere un comentario, no me quedó todo lo completo que requiere el tema. Lo que me gustaría explicar daría para muchas horas de charlas. De hecho, enseño estos temas en mi asignatura de Sistemas Operativos (II), dedico al menos unas 12 hs de clase, y aún así no entramos en muchos detalles importantes. Pero intentaré resumirlo en este apunte, fundamentalmente para que se entiendan los problemas de arquitectura, y de cómo toda decisión que se tome en una arquitectura, lenguaje o programa tiene implicaciones positivas y negativas, siempre.

Lo básico

Apple tomó muchas decisiones técnicas con el objetivo de mejorar la “experiencia de usuario”, pero por sí mismas serían inútiles -o más perjudiciales- si no van acompañadas de medidas de control estrictas. Es fundamental el papel de los controles de las aplicaciones en el App Store (¡ojo! no los justifico). Independientemente de otras consideraciones políticas, dado su simplicidad y soluciones ad hoc, el iOS sería muy fácil de “abusar” por los desarrolladores y aplicaciones.

Por el contrario, Android es una plataforma abierta que puede usar cualquier fabricante, que permite la instalación de cualquier aplicación. El control de aplicaciones del Market por parte de Google es prácticamente inexistente. Esto hace que no valgan soluciones ad hoc, obliga a implementar en el sistema operativo medidas de seguridad y control “canónicas”, en el sentido que mantenga las condiciones fundamentales de todo sistema operativo de propósito general: eficiencia, equidad (fairness) y seguridad.

Cualquiera que haya estudiado la teoría y arquitectura de sistemas operativos (o que haya leído sobre el tema) entiende perfectamente que la eficiencia, equidad y seguridad son objetivos contradictorios. Mayor equidad o seguridad afectan negativamente a la eficiencia, y viceversa.

Hay otro problema importante. Se desea que los sistemas operativos de teléfonos actúen como sistemas de tiempo real también para las aplicaciones (ya tienen que serlos en otros aspectos, como la “radio” a interacción con protocolos de red de telefonía).  Es decir, con límites precisos de “tiempo de respuesta”: el tiempo que transcurre desde que ocurre un evento (como tocar la pantalla) hasta que se empieza a ejecutar al “gestor” de ese evento (la aplicación).

Los sistemas Unix (como Linux o Darwin) no son de tiempo real, su planificador de procesos (scheduler) es no-determinístico. No lo es por una sencilla razón: si se pretende una buena “multiprogramación” (llamada comunmente “multitarea”) se deben cumplir los objetivos de equidad y otros más complejos, como evitar los tiempos de espera muy largos (starvation). Esos dos objetivos impiden asegurar (formalmente) que el sistema sea de tiempo real. Además, los sistemas de tiempo real exigen que el tiempo de ejecución de cada ráfaga de ejecución de las aplicaciones (CPU bursts) críticas sea lo suficientemente breve para poder asegurar “tiempo real” a las demás aplicaciones (recordad que el scheduler debe ser determinístico, por ende no puede asegurar equidad).

En pocas palabras, si deseas comportamientos próximos al tiempo real necesitas relajar los requerimientos de un sistema de propósito general, y aumentar el control de las aplicaciones que se ejecutan en ese sistema.

¿Se entiende el balance y decisiones contradictorias que hay que tomar? En un sistema de multiprogramación, con el hardware equivalente (los dispositivos iOS y Android lo son), si  se desea disminuir las latencias se deben relajar otras condiciones del planificación de procesador (por lo que se obtiene una “peor multitarea”) y aumentar el control de aplicaciones (lo que genera otros problemas políticos, éticos, sociales y de negocio). Si pretende más apertura y libertad, se debe ser muy estricto con equidad y seguridad, lo que genera efectos negativos sobre el tiempo de respuesta.

Al final el tema de fondo no es [sólo] técnico, sino de otros aspectos más filosóficos: control vs apertura.

Creo que ya expliqué la base del problema técnico-filosófico, ya puedes dejar de leer si te agobian los detalles técnicos. Pero si te interesan, intentaré resumirlos en las cuatro partes que tienen más responsabilidad en los “tiempos de respuesta” de un sistema: las aplicaciones, el lenguaje y plataforma, la gestión de memoria, y el scheduler.

Las aplicaciones

El tipo de programación para móviles es esencialmente “dirigida por eventos” . Cuando el usuario toca la pantalla se llama la función que debe responder a ese evento (en Android se llaman “actividades”). La recomendación es que la aplicación no necesite mucha CPU ni tiempo para generar la respuesta, especialmente en los menús principales. Por supuesto, a este nivel, todo depende de los programadores de la aplicación, y supongo que estos son los mayores culpables de la “reacción lenta” de los menús cuando se interactúa con una aplicación.

Las aplicaciones también pueden tener otros malos comportamientos, por ejemplo consumir mucha CPU (aún cuando están en background o como servicio), no sólo molesta a todos los demás procesos, también aumenta el consumo de batería y la temperatura.

¿Cómo solucionar este problema? O se hace un control estricto de cada aplicación, o se implanta un scheduler más sofisticado y genérico que puede tener algunas aristas.

Ya sabéis qué decisión se tomó en Apple, y cuál en Google.

El lenguaje y plataforma

iOS y Android difieren mucho en este aspecto.iOS usa el mismo lenguaje estándar del Mac OS X, el Objective-C. Los programas se compilan directamente en código ejecutable para el procesador. En cambio Android optó por usar Java, que se convierte a un lenguaje intermedio ejecutado por una máquina virtual optimizada (similar la máquina virtual de Java, pero es diferente), Dalvik.

Obviamente el rendimiento entre ambas es muy diferente, el código ejecutable siempre será más eficiente que otro se que no se ejecuta directamente en el procesador. Pero esto no significa que los diseñadores de Android sean unos imbéciles, de nuevo, fueron decisiones estratégicas que están muy por encima de temas técnicos, aunque terminan afectando dramáticamente a este último.

Si el sistema operativo se va a ejecutar siempre sobre una única arquitectura, como hizo históricamente Apple (aunque tuvo dos migraciones importantes, exitosas pero dolorosas y muy molestas para los desarrolladores), la opción natural es generar directamente el ejecutable. Además, Apple licenció el diseño de ARM [*] y compró una empresa de diseño de chips (Intrinsity) para que se encargue del diseño de sus procesadores y no depender ni de fabricantes externos (antes lo hacía Samsung).

Por otro lado, la intención de Android era entrar a competir en el mercado con un sistema libre/abierto y multiplataforma (después de varios intentos fallidos, como OpenMoko), por  lo que era preferible un sistema que no obligase a compilar una aplicación para cada tipo de procesador. Si no, estaría condenada al fracaso. Para solucionar el problema de la eficiencia diseñaron desde cero el Dalvik, que al usar el lenguaje Java ya tenía a su disposición muchas herramientas (como Eclipse), librerías de desarrollo (como las clases estándar de Java), y una comunidad enorme de programadores (me abstengo de opinar sobre Java, pero afortunadamente Android lo ha mejorado muchísimo con su API y framework de programación).

Aunque Java ya está preparado para multithreading, por cuestiones de seguridad los diseñadores de Android prefirieron, muy acertadamente, ejecutar cada aplicación como un proceso diferente y completamente aislado. Dejaron esta parte de la gestión de procesos al núcleo Linux, que es muy eficiente y está más que probado.  Con esta arquitectura de máquinas virtuales replicadas en procesos independientes, se generan otras infeficiencias: cada proceso debe cargar su máquina virtual, y éste debe cargar después todas las clases estándar de Java que hacen falta para la aplicación. Esto haría que el tiempo de arranque de cada programa fuese inaceptable, y han recurrido a un truco muy guapo para solucionarlo (por eso digo que la arquitectura de Android es una obra impresionante de ingeniería informática): el zygote.

El kernel Linux implementa un mecanismo llamado copy-on-write ( COW) que permite que los procesos hijos (creados con el fork()) compartan memoria con el padre (por eso la creación de procesos es muy rápida de Linux). Cuando un proceso crea un hijo, éste reusa la misma memoria y tablas de páginas del padre, pero todas las páginas se marcan como “sólo lectura”. El hijo puede comenzar a ejecutarse muy rápidamente. Cuando uno de los procesos intenta escribir sobre una página compartida, se genera una interrupción de error (trap), el sistema operativo coge el control, verifica que se trata de memoria COW, asigna una nueva página, copia el contenido de la página afectada, modifica la tabla de páginas de uno de los procesos para apuntar a esta nueva, y permite la continuación de ambos procesos. Además de la rapidez con que puede crear y ejecutar procesos, se ahorra mucha memoria RAM, por ejemplo, el código ejecutale y las librerías comunes siempre se comparten (nunca se modifican).

¿Cómo aprovecha Android este mecanismo? Crea un proceso inicial, el zygote, que carga la máquina virtual y las librerías estándares de Java y del API de Android. Cada nueva aplicación que se arranca es hija de este zygote, et voilà, ya ganaron eficiencia y se ahorra muchísima memoria (Chrome y Chromium usan la misma técnica para abrir pestañas y ventanas rápidamente).

¿Es o no es una obra maestra de ingeniería? Claro que sí, se han tomado todo ese trabajo para compensar el problema de eficiencia y seguridad de una máquina virtual. Por eso, cuando pienso que por debajo hay una máquina vrtual ejecutando el código, me maravillo con la velocidad de ejecución de las aplicaciones en Android.

Por supuesto, esta arquitectura con memoria virtual tiene otros efectos negativos a la hora de medir los tiempos de respuesta: en la gestión de memoria y cache, que trato más adelante.

[*] En la biografía de Steve Jobs cuentan como después de evaluar usar Intel para los iPad, Jobs le dijo muy cabreado a los ejecutivos de Intel algo así como: “son muy buenos para procesadores de alto rendimiento, pero sois unos inútiles para procesadores de bajo consumo”.

La gestión de memoria del núcleo

Los procesadores que usan iOS y Android son similares en cuanto a su gestión de memoria (también similares a los ordenadores que usáis). Ambos gestionan memoria virtual con paginación. Los procesos no generan direcciones físicas de RAM, sino direcciones lógicas que son convertidas a direcciones físicas en cada ejecución por el módulo conocido como Memory Manager Unit (MMU). Cada proceso tiene sus propias tablas de página que mantienen la relación de número de página del proceso (es el índice a la tabla) a ubicación en la memoria RAM. Los procesadores ARM permiten páginas de 4KB a 64KB (lo usual son 4KB), así que cada aplicación tiene miles de páginas, y por lo tanto tablas de miles de entradas (en ARM hay tablas de dos niveles, en la arquitectura Intel o AMD, cuatro niveles).

Cada entrada de la tabla almacena propiedades de la página: lectura (r), escritura (w), ejecución (x), accedida (a, o acces bit), modificada (d, o dirty bit), cargada en memoria (p, present bit), etc. Por esta razón las tablas se mantienen en la memoria RAM, lo que genera otro problema ¿cómo hace el MMU para no tener que acceder dos veces a la memoria -primero a la tablas de página y luego a la dirección que se pretende acceder-? Sin un mecanismo que lo solucione, el tiempo efectivo de acceso a la memoria sería al menos el doble de lo que permite la memoria RAM (o sea, una patata de lento).

Para solucionar este problema se usan sistemas de cache que almacenan en registros del procesador un conjunto de entradas de las tablas: el translation-lookaside-buffer (TLB). El TLB de los ARM suelen tener 128 entradas, cada una de ellas es una copia idéntica de la entrada de la tabla de páginas. Se intenta tener en el TLB las entradas más usadas, así, cada vez que el MMU encuentra la entrada en el TLB puede acceder directamente a la memoria (hit), si no, tiene que descartar una entrada y cargar la que hace falta (fault). El hit ratio es una indicación de la velocidad de acceso efectivo a memoria, viene dado por el procesador y el algoritmo de sustitución que se use, a mayor número de entradas en el TLB, el hit ratio sube.

Cada vez que el scheduler selecciona otra aplicación para ejecutar se produce un cambio de contexto, entre otras cosas significa que las entradas del TLB deben ser invalidadas (ya no sirven para el nuevo proceso, por seguridad tampoco debe poder usar esos datos), además las entradas que han sido modificadas por el procesasor (por ejemplo, que una una página fue accedida o modificada) deben ser copiadas a la tabla original en memoria (flush). Por eso se dice que los cambios de contexto son “caros”, y es muy imporante las decisiones que toma el scheduler para minimizar los cambios de contexto “inútiles” (¡pobre el scheduler!, las cosas que tiene que tener en cuenta para que lo usuarios no se quejen porque el audio pega saltos).

Un proceso con máquina virtual como Android mete mucha más presión al TLB (y por ende baja el hit ratio) que un proceso que ejecuta código nativo. Para ejecutar una función equivalente en máquina virtual se necesita más memoria, la del código del intérprete, la memoria usado por éste, más el código en lenguaje intermedio y la memoria que necesita el programa. No sólo eso, prácticamente cada ejecución del código intermedio implica cambios en las variables del propio programa, también en las de la máquina virtual, forzando a que en cada cambio de contexto sean más las entradas del TLB que deben ser copiadas a la RAM. O sea, los cambios de contexto son también más caros.

Lo mismo que pasa con el TLB (recordemos que es un tipo específico de cache), pasa con la memoria de cache de RAM en el procesador, la máquina virtual mete más presión, por lo que también bajan sus hit ratios.

De nuevo, una decisión de negocio (y política y filosófica), genera muchas complicaciones técnicas que hay que resolver. Si uno las tiene en cuenta, empieza a dejar de pensar en términos de fanboy y se cuestiona muchas cosas. Entre otras, lo eficiente que tiene que ser Linux para gestionar esta complejidad adicional sin que se note demasiado la diferencia.

El scheduler

Se llama scheduler al módulo del sistema operativo que selecciona cuál será el siguiente proceso a ejecutar. A cada proceso se le le asigna un tiempo máximo de ejecución: el cuanto (o quantum). Si al proceso se le acaba el cuanto, el sistema operativo se apropia de la CPU (por eso se llaman apropiativos o preemptive), y llama al scheduler para seleccionar el siguiente de la lista de procesos que esperan para ejecutar (este mecanismo se denomina round robin). El scheduler de Linux (como la gran mayoría) no son determinísticos, significa que a priori no se puede conocer la secuencia de ejecución de procesos, por lo que no se puede asegurar analítica o formalmente que los procesos se ejecuten en un tiempo máximo predeterminado (si éste es suficientemente pequeño).

No todos los procesos tienen los mismos requerimientos, aquellos interactivos, o de reproducción multimedia necesitan menores tiempo de respuesta que un servicio en background, o que consume mucha CPU. Para eso se usan las prioridades, cada proceso tiene una prioridad inicial (por ejemplo “multimedia”, “normal”, “de fondo”, etc.), luego el scheduler la ajusta dinámicamente (prioridad dinámica) dependiendo del comportamiento del proceso y las necesidades de los otros. Por eso, el comportamiento de un sistema que use prioridades depende del comportamiento global de todos los procesos, estos no se pueden predecir, y de allí que sea no determinístico, y que se generen tiempos de respuesta variables.

El scheduler  además tiene que tomar en cuenta muchos factores adicionales (p.e., minimizar los cambios de contexto), y evitar que se produzcan esperas muy largas (starvation) que se pueden producir porque procesos con mayor prioridad están cogiendo siempre la CPU y no dejan ejecutar a otros de menor prioridad. Imaginaros que tenéis el reproducto de música de fondo, y que estáis por sacar una foto. La aplicación de foto podría estar consumiendo el 100% de CPU para actualizar la imagen, por lo que produciría cortes en la música, insoportablemente molesto ¿no? Pues es el pobre scheduler el que tiene que evitar estos problemas, tomando decisiones muy rápidas, cientos de veces por segundo.

Para evitar estas esperas tan largas en los casos extremos (corner cases) como el que acabo de decir, el scheduler mantiene dos colas, la activa y la inactiva. Siempre asigna procesos desde la cola activa. A cada proceso se le asigna un tiempo máximo (además del cuanto) que puede estar en la cola activa (el timeslice, por ejemplo 500 mseg), cuando consume todo su tiempo se le mueve a la cola inactiva. Cuando ya no queda nadie en la cola activa, el scheduler cambia, la activa pasa a ser la inactiva, y viceversa. Esto aumenta aún más el efecto “no determinístico” del scheduler, y puede producir saltos o latencias elevadas si el procesador está temporalmente sobrecargado.

Para evitar estas latencias que molestan a la interactividad del usuario se pueden tomar varias medidas ad hoc, por ejemplo subir la prioridad al máximo a la aplicación que está activa (así hacía Windows), pero esto genera otros problemas: ¿si la aplicacion activa abusa y no deja de usar la CPU? (un truco obvio de programador para que su aplicación parezca que es más rápida y eficiente ¿no?) ¿y si hay procesos en background que necesitan la CPU  -por ejemplo un SMS, o respondera la red antes que pase el timeout- y esperan demasiado tiempo?

Pues es muy complicado, y cualquier “truco” que se ponga al scheduler deja la puerta abierta para que los programadores abusen… salvo que hagas un control exhaustivo de cada aplicación que se vaya instalar en el teléfono. ¿Se entiende por qué Apple puede recurrir a trucos ad hoc y evitar al mismo tiempo el abuso de las aplicaciones?

Si por el contrario renuncias a hacer ese control, no queda más remedio que el scheduler haga su trabajo, como en cualquier sistema operativo de uso genérico. Y que la solución adoptada sea lo suficientemente buena, simple, genérica y probadamente segura. ¿Se entiende por qué la solución en el Linux de Android es más difícil de encontrar? ¿o por qué le llamé una “solución canónica?

¿Como puede Android mejorar los tiempos de respuesta?

Evidentemente hay dos caminos obvios.

Por un lado, mejorar paulatinamente el scheduler (si es que allí reside el problema). No es una cuestión de un día, ni un año, en Linux se tardó años en encontrar un planificador que funcione tan bien como la actual para sistemas interactivos (el scheduler CFQ).

La otra parte, la gestión de memoria, depende en gran medida del apoyo del hardware. Un procesador con más capacidad de cache y TLB mejoraría mucho, pero también tiene su coste: más transistores, más consumo, más temperatura, mayor tamaño. A medida que se mejore el proceso de fabricación (más transistores en el mismo chip), seguramente incluiran TLB y cache con más capacidad. Lo mismo ocurre con el aumento de la potencia bruta y el aumento de cores (aunque  creo que afectan en mejnor medida a los tiempos de respuesta de la interfaz).

Tampoco me extrañaría que el código de los drivers de la pantalla sea de baja calidad (como suelen ser todos los drives, comparados con el núcleo del sistema operativo, se hacen bajo presión, y la prioridad es el hardware, no el SO) y encuentren big locks que no tocan.

De todas formas, los tiempos irán mejorando paulatinamente hasta que no podamos detectar diferencias en las latencias de la interactividad. Al mismo tiempo, iOS tendrá que ir incluyendo mejores capacidad de “multitarea” (como ya pasó desde que salió la primera versión de iOS, y seguramente irá reduciendo los controles de calidad que hace a cada aplicación).

Supongo que os dáis cuenta de la complejidad técnica de estos “cacharritos”, y que las decisiones de negocio imponen muchas restricciones y problemas al desarrollo del software, lo que obliga a los ingenieros a aplicarse a fondo. No es que los ingenieros de iOS son unos genios, y los de Android unos inútiles, los requerimientos y grados de libertad son diferentes.

Ley informática: optar por la apertura tiene costes técnicos iniciales, optar por el software libre tiene costes técnicos iniciales, pero producir una plataforma totalmente bajo control también tiene costes (sociales), y a más largo plazo.

Desarrollo Front End : directrices

[Fuente: http://taitems.github.com/Front-End-Development-Guidelines/#]

Accesibilidad

What’s Up, DOCTYPE?

The absence of a DOCTYPE is a crime punishable by death. You may have relied on the following DOCTYPE in the past, but it’s important to know that this is now being superseded by a leaner and meaner snippet.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
	"http://w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

Ideally, the HTML5 DOCTYPE should be used. It’s supported in all modern browsers, and throws IE6 and IE7 into standards modeSource.

<!DOCTYPE html>

Write Valid Semantic Markup

Writing websites with clean, semantic HTML is something we wish we could always do. Sometimes we find ourselves limited by the way pages were setup by our predecessors, or sometimes we’re coding an HTML email. The validity of the HTML should never be compromised, even if to solve a browser specific bug.

Headings should be heirarchically created from <h2> onwards, paragraphs should always be in <p> tags and so on and so forth. If you write semantic HTML, the resultant page will be cleaner, lighter and easily parsed by search engine spiders. This is one of the simplest SEO fixes you can undertake.

WHICH DO YOU THINK LOOKS CLEANER, THIS?:

<span class="sectionHeading">A Heading</span>
<br /> <br />
Lorem ipsum dolor sit amet. ...
<br /> <br />

OR THIS?

<h2>A Heading</h2>
<p>
	Lorem ipsum dolor sit amet. ...
</p>

Fallbacks for Middle Mouse Clicks

One of the most frustrating accessibility and usability flaws of the modern web stems from the remapping of hyperlink click functions. Elements that appear to be hyperlinks may have their single click functionality remapped via JavaScript, breaking middle mouse click (open in new tab) functionality. If they can be opened in a new tab, their href of a single hash sends you back to the same page.

A modern example of a popular website that is contributing to this problem is the Twitter web app. Middle mouse clicking of names or user avatars yields completely different results throughout the web app.

<!-- The old way, breaking the web -->
<a href="#"></a>

<!-- If you can't deliver a page on mouse click, it's not a hyperlink -->
<span class="link" role="link"></span>

Another alternative is the use of “hashbangs“, that remap normal URLs to hash links and fetch pages via AJAX. Libraries that provide hashbang functionality should be able to display the page normally when middle mouse clicked, or load the content from that page into a designated area when clicked normally. But tread carefully, there are plenty of people who believe hashbangs are breaking the web.

Use Microformats

Microformats are a way of making contact information machine readable. hCard classes (not vCard) are used to define the type of content contained within elements. These are then extracted or highlighted by the browser.

<span class="tel">
	<span class="type">home</span>:
	<span class="value">+1.415.555.1212</span>
</span>

If you were to navigate to a page that uses this, you would notice that a program like Skype will easily detect what numbers on the page are phone numbers. Mobile Safari does something similar on iOS devices.

For more information: http://microformats.org/wiki/hcard

Images Need ‘Alt’ Text

The <img> tag requires alt text to both validate and meet accessibility guidelines. The text in the alt attribute should be descriptive of what the image shows, or is trying to achieve, unless of course the image is not critical.

If the image is of a list bullet or other trivial icons, it is recommended to simply leave the alt attribute empty, but still present. A screenreader will then ignore it, as opposed to having to read out “bullet” 20 times.

<img src="dog.gif" alt="Fido and I at the park!" />
<!-- good, descriptive -->

<img src="bullet.gif" alt="bullet" />
<!-- bad, as silly as it seems -->

<img src="bullet.gif" alt="" />
<!-- good -->

Use Tables for Tabular Data Only

Tables should only ever be used for the presentation of tabular data. The only exception is when composing HTML email, in which a table is almost the only thing supported by soul crushing email clients.

For accessibility, table headers should always be presented using <th> elements. Remember to also set cellpadding,cellspacing and border values to 0 as these are more consistently controlled by CSS.

<table cellpadding="0" cellspacing="0" border="0">
	<thead>
		<tr>
			<th>
				Cell Header
			</th>
		</tr>
	</thead>
	<tbody>
		<tr>
			<td>
				Cell Item
			</td>
		</tr>
	</tbody>
</table>

Use jQuery & jQuery UI Widgets

jQuery and jQuery UI are constructed to look and behave as close to identical as possible on different browsers. jQuery UI is designed to be WAI WCAG 2.0 and WAI ARIA compliant, so using the framework removes any uncertainty about plugins or scripts running on your site.

JavaScript

Whitespacing & Formatting

Any discussion about formatting, whitespacing and the placement of braces is going to be hotly debated. I guess the simplest rule is that, unless you’re willing to completely format a whole document, respect and maintain the formatting of an existing document. That means: see same-line braces throughout a JS file, continue to write code with same-line braces. Your code should fail the code review process if it doesn’t maintain consistency with the rest of the document.

Consistent formatting makes code more readable, and also means the code can be easily modified with find and replace commands. The coding habits we have picked up are thankfully very similar to what jQuery officially encourages. There are a few minor discrepencies, but again, these are personal issues or things that we think cannot be maintained. Further Reading

CHARACTER SPACING

// Bad
if(blah==="foo"){
	foo("bar");
}

// Good :)
if (blah === "foo") {
	foo("bar");
}

SAME LINE BRACES

// Bad
if (foo)
{
	bar();
}

// Good :)
if (foo) {
	bar();
}

ALWAYS USING BRACES

// Bad
if (foo)
	bar();

// Good :)
if (foo) {
	bar();
}

STRING HANDLING

Strings should always use double quotes. Some people are very fond of their C style strings (single quotes), but this leads to conflicting styles within a script. C style string handling dictates that empty and single character strings should be wrapped in single quotations, while phrases and words should be wrapped in double quotations.

Commenting

The requirement to comment code obsessively was pioneered by managers, team leaders and other people that interact with code infrequently. It is sought merely as a check box for an employee’s KPIs, and provides little return for the time spent doing so.

If a best-practice oriented developer follows the guidelines established in this document, their code should become so readable and obvious that the need to comment what it is doing is embarassingly redundant. Consider the following example. In this: booleans are posed as questions, and functions are named intuitively.

if (user.hasPermission) {
	editPage();
}

Commenting, in this scenario at least, is completely unnecessary.

SITUATIONS WHERE COMMENTING IS IMPORTANT

Some parts of a project will never be easy to scan and understand. Consider a complicated regular expression, or a math function calculating angles or switching between degrees and radians. Without the comment above, beginner and intermediate readers will be fairly clueless to the scripts’ meaning.

// RegEx for validating US phone numbers, can be (XXX) XXX-XXXX (with or without dashes, spaces or brackets)
var phoneRegEx = /^(?(d{3}))?[- ]?(d{3})[- ]?(d{4})$/;

Always Use === Comparison

The use of the == equality operator allows for frustrating bugs to slip through almost undetected. It allows for weak typing that is best explained by JavaScript Garden. The use of the strict equality operator === does not run type coercion and therefore strictly evaluates the difference between two objects. Again, consult JavaScript Garden for more information

var zeroAsAString = "0";

if (zeroAsAString == 0) {
	// gets in here lolwut
}

if (zeroAsAString === 0) {
	// never gets in here
}

THE EXCEPTION

Double equals comparison is allowed when comparing to null, because it will detect both null or undefined properties. If you don’t fully understand this, I still suggest you use triple equals.

var foo = null;

// foo is null, but bar is undefined as it has not been declared
if (foo == null && bar == null) {
	// still got in here
}

Always Specify the Second ‘radix’ Parameter When Using .parseInt()

When parsing a string to an integer, it is considered good practice to specify the second ‘radix’ parameter – which determines to what base the string should be converted to. The default setting will trigger a radix of 16 whenever the string is lead by a 0. Most beginner and intermediate users are only ever going to be using a radix of 10. Thanks to João Moreno for logging thecorrection.

alert( parseInt("08") ); // alerts: 2

alert( parseInt("08", 10) ); // alerts: 8

Avoid Comparing to true and false

Direct comparison to the values of true and false is unnecessary. Sometimes it might be good for clarity, but it’s just extra code.

if (foo === true) {
	// good that they're using triple equals, bad as it's redundant
}

if (foo) {
	// yay!
}

if (!bar) {
	// the opposite
}

Avoid Polluting the Global Namespace

An over-reliance on global variables is something all of us, myself especially, are guilty of. Arguments as to why globals are bad are fairly straight forward: the chance of script and variable conflicts is increased, and both the source file and the namespace itself become littered with countless ambiguously named variables.

Douglas Crockford believes that the quality of a JavaScript application can be assessed by the number of global variables it uses; the less the better. Given that not everything can be a local (but let’s be honest, that one you’re thinking about right now, it can, don’t be lazy) you need to find a way of structuring your variables to prevent clashes and minimise the bloat. The easiest way is to employ a single variable or a minimal amount of modules on which the variables are set. Crockford mentions that YUI uses a single global, YAHOO. He discusses this in more detail in his blog post “Global Domination”.

Considering that, in the case of small web apps, globals are generally used to store application-wide settings: it’s generally better to namespace your project or settings as objects.

// polluted global name space
var settingA = true;
var settingB = false;
var settingC = "test";

// a settings namespace
var settings = {
	settingA: true,
	settingB: false,
	settingC: "test"
}

But if we’re avoiding globals to reduce the chance of conflicts, isn’t standardising the namespaces to be the same going to increase chance of one app’s settings overwriting anothers? Well, it would make sense. It is instead suggested that you namespace your globals to your own specific app name, or reassign your namespace much in the same way that jQuery uses$.noConflict() mode.

var myAppName = {
	settings: {
		settingA: true
	}
}

//accessed as
myAppName.settings.settingA; // true

Camel Case Variables

The camel casing (or camelCasing) of JavaScript variables is accepted as the standard in most coding environments. The only exception that was raised in the comment section is the use of uppercase and underscores to denote contants.

var X_Position = obj.scrollLeft;

var xPosition = obj.scrollLeft; // tidier

SCENE_GRAVITY = 1; // constant

Loop Performance – Cache Array Length

Looping is arguably the most important part of JavaScript performance to get right. Shave a millisecond or two off inside of a loop, potentially gain seconds overall. One such way is to cache the length of an array so it doesnt have to be calculated every time the loop is iterated through.

var toLoop = new Array(1000);

for (var i = 0; i < toLoop.length; i++) {
	// BAD - the length has to be evaluated 1000 times
}

for (var i = 0, len = toLoop.length; i < len; i++) {
	// GOOD - the length is only looked up once and then cached
}

THE EXCEPTION

If you’re looping through an array to find an remove a particular item, this will alter the array length. Any time you change the array length by either adding or removing items from inside the loop, you will get yourself into trouble. Consider either re-setting the length or avoid caching it for this particular situation

Loop Performance – Use ‘break;’ & ‘continue;’

The ability to step over and out of loops is really useful in avoiding costly loop cycles.

If you’re looking for something inside of a loop, what do you do once you find it? Say the condition you’re looking for is matched halfway through a 1000 item loop. Do you execute whatever you intend to do, and allow the loop to continue to iterate over the remaining 500 items, knowing that there’s no chance it will hit an if statement? Nope! You break out of your loop, literally!

var bigArray = new Array(1000);

for (var i = 0, len = bigArray.length; i < len; i++) {
	if (i === 500) {
		break;
	}
	console.log(i); // will only log out 0 - 499
}

Another problem is skipping over a particular iteration and then continuing on with the loop. While things like odds and evens are better managed by replacing i++ with i + 2, some conditions need to be specifically listened for, to then trigger the skip. Anything that prevent’s running through an entire iteration is pretty handy.

var bigArray = new Array(1000);

for (var i = 0, len = bigArray.length; i < len; i++) {
	if (condition) {
		continue;
	}
	doCostlyStuff();
}

Don’t Send Too Many Function Parameters

This is a pretty bad idea, more for readability than anything:

function greet(name, language, age, gender, hairColour, eyeColour) {
	alert(name);
}

It’s a much better idea to construct an object before-hand or to pass the object inline

function greet(user) {
	alert(user.name);
}

greet({
	name: "Bob",
	gender: "male"
});

Remap ‘this’ to ‘self’

When writing object-oriented (OO) JavaScript, the scope of this must be understood. Regardless of what design pattern you choose to structure your pseudo-classes, a reference to this is generally the easiest way to refer back to an instance. The moment you begin integrating jQuery helper methods with your pseudo-classes is the moment you notice the changing scope of this.

Bob.findFriend("Barry");

Person.prototype.findFriend = function(toFind) {
	// this = Bob
	$(this.friends).each(function() {
		// this = Bob.friends[i]
		if (this.name === toFind) {
			// this = Barry
			return this;
		}
	});
}

In the above example, this has changed from a reference to Bob, to his friend Barry. It’s important to understand what happened to the value of this over time. Inside of the prototyped function, this refers to the instance of the pseudo-class (in this case Bob). Once we step inside the $.each() loop, this is then re-mapped to be item i in the parsed array.

The solution is to remap the value of this to either self or _self. While self (sans underscore) is not exactly a reserved keyword, it is a property of the window object. Although my use of self was picked up from the jQuery source code, they have realised their mistake and are attempting to rectify the situation and instead use _self. Personally, I prefer the use of self for the sheer cleanliness – but it can throw some pretty confusing bugs for people. Tread carefully.

In the following example I will better utilise the parameters made available with the $.each() helper, as well as re-mapping the value of this.

Bob.findFriend("Barry");

Person.prototype.findFriend = function(toFind) {

	// the only time "this" is used
	var _self = this; 

	$(_self.friends).each(function(i,item) {
		if (item.name === toFind) {
			return item;
		}
	});

}

CanIHaz Boolean?

Booleans should be easily identifiable by the way they are named. Use prefixes like iscan or has to propose a question.

isEditing = true;

obj.canEdit = true;

user.hasPermission = true;

Minimising Repaints & Reflows

Repaints and reflows relate to the process of re-rendering the DOM when particular properties or elements are altered. Repaints are triggered when an element’s look is changed without altering its layout. Nicole Sullivan describes these changes in a thorough blog post as style changes such as visibility or background-color. Reflows are the more costly alternative, caused by changes that alter the layout of the page. Examples include the addition or removal of elements, changes to an element’s width or height, and even resizing the browser window. Worst yet is the domino effect of reflows that cause ancestor, sibling and child elements to reflow.

There is no doubt that both reflows and repaints should be avoided if possible, but how?

A REFLOW EXAMPLE

It’s not that the following snippet is “bad code” exactly. But let’s assume that the array arr has 10 items.

var myList = document.getElementById("myList");

for (var i = 0, len = arr.length; i < len; i++) {

	myList.innerHTML += "<li>" + arr[i].title + "</li>"; //reflow - appending to element

}

In the above for loop, a reflow will be triggered for every iteration of the loop. 10 iterations cause 10 reflows.

Now consider the following:

var constructedHTML = "";

for (var i = 0, len = arr.length; i < len; i++) {
	constructedHTML += "<li>" + arr[i].title + "</li>"; //no reflow - appending to string
}

document.getElementById("myList").innerHTML = constructedHTML; //reflow

In this scenario, the elements are being constructed within a string. Not a single reflow is created by the loop, as the DOM is not being altered. Only once the array has been completely looped through is the string then applied as the innerHTML of an object, causing the only reflow of the function.

There are endless types of reflows and repaints that can be avoided, and lucky you gets to go on an read about them. Reading material on the subject matter is plentiful, but most of it is linked to from the excellent starting point that is Nicole Sullivan’s blog post. There are important lessons to be taken away from this when it comes to a multitude of technologies synonymous with “web 3.0” and HTML5. The lesson above can be directly applied to writing jQuery. It’s also important to consider when fiddling with canvas, and trying to keep a frame rate in the 30-60 range.

Don’t Use Milliseconds to Generate Unique IDs

There is a method for generating unique IDs that has hung around since the early days of web dev. It involved appending the elapsed milliseconds since January 1, 1970 to your static ID by way of:

var myID = "static" + new Date().getTime();

This was a fairly foolproof method originally, because even if two ofthe above lines were performed one after the other, a few millisecondsnormally separated their execution. New browsers brought with them newJavaScript engines, coupled with ever increasing clock speed. Thesedays it’s more likely that your milliseconds match than are slightlyincremented.

This leads to bugs that are near impossible to debug by conventionalmeans. Because your DOM is created on the fly, traditional validationof the page source won’t identify multiple IDs as an error. JavaScriptand iQuery error handling dictates that the first match for the IDwill be utilised and other matches ignored. So it doesn’t even throw aJS error!

No, the only real method to debug it is line by line breakpoint ingand logging – but “pause” at the wrong line and your milliseconds willno longer clash!

The good thing is that there are plenty of alternatives. To be pedantic, it’s worth noting that a computer’s random function is not truly random as it is seeded by system time- but the probability of clashes is rather minuscule.

var myID = "static" + Math.round(Math.random() * 10000);

Personally, I’m partial to a bit of faux GUID generation. Technicallya GUID is generated according to your hardware, but this JavaScriptfunction does the next best thing. The following is a handy function I’ve pinched from a stack overflow post.

function S4() {
   return (((1+Math.random())*0x10000)|0).toString(16).substring(1);
}
function guid() {
   return (S4()+S4()+"-"+S4()+"-"+S4()+"-"+S4()+"-"+S4()+S4()+S4());
}
var myID = "static" + guid();

Feature Sniff, Don’t Browser Sniff

Does the client’s browser support geolocation? Does the client’s browser support web workers? HTML5 video? HTML5 audio? The answer used to be:

if ($.browser.msie) {
	// no it doesn't
}

But things are rapidly changing. The latest version of IE is almost a modern browser, but as usual it’s making front end development a pain. Earlier versions of IE were generally as equally sucky as their predecessors, so it enabled lazy JavaScript developers to simply detect if (ie) and execute some proprietary Microsoft slops syntax. Now IE9 has done away with these functions, but that old if (ie) chestnut is throwing a spanner in the works.

So what if you could detect support for individual features without sniffing the (unreliable and cloakable) user-agent?

If you answered “that would be ghetto”, then you are correct.

In steps Modernizr, a JavaScript library developed in part by industry dream-boat Paul Irish. With wide adoption, tiny file-size and plenty of documentation: implementing it is a no-brainer. It creates a Modernizr object that contains the results of its detection tests, so checking feature support is as simple as the following:

// old way of detecting canvas support
if (!!document.createElement('canvas').getContext) { ... }

// with Modernizr
if (Modernizr.canvas) { ... }

Readable Milliseconds

A handy way of writing milliseconds in a readable format. Great for beginners, but mostly a gimmick.

// is this 3, 30 or 300 seconds?
var timeout = 30000; 

// an extra calculation, but easier to read and modify
var timeout = 30 * 1000;

jQuery Specific

Chain Like a Mad Dog

One of the best parts of jQuery is its function chaining. You’ve probably used it a bit, maybe a few simple calls one after another… but have you ever traversed the DOM like a mad dog? Take some time to familiarise yourself with the .end() function. It is critical for when you begin stepping up and down the DOM tree from your original selector.

$(".quote")
	.hide()
	.find("a").text("Click here").bind("click",doStuff).end()
	.parent().removeClass().addClass("testimonial").draggable().end()
	.fadeIn("slow");

In the example above, the .end() function is used once we have finished doing things with a particular DOM object and want to traverse back up the DOM to the original object we called. We then load back up and dive back into the DOM.

Using data-* Attributes

Those of you who have been writing JavaScript (and not jQuery) for a good length of time are most likely familiar with attributes. Setting them. Getting them. Abusing rel and title instead…

So when isn’t HTML5 or jQuery coming the rescue? New specs allow the use of data- prefixes on HTML elements to indicate attributes which can hold data, and jQuery does an awesome job of converting the designated string into the correct JavaScript type. It’s a beautiful partnership. Let’s create a DIV with some data attributes.

<div id="test" data-is-bool="true" data-some-number="123"></div>

Now even though our values are wrapped in quotation marks, they won’t be handled as strings:

typeof $("#test").data("isBool"); // boolean

typeof $("#test").data("someNumber"); // number

SPECIAL CASING

It’s also important to notice the lower casing required to get these snippets to work. But if you’re a great front end developer, you will still want to camel case your data variables. Like many places in JavaScript, a preceding hyphen signifies camel casing of the next letter. The following camel casing of the HTML attribute does not work and the same JavaScript used above will return undefined.

Does not work 🙁

<div id="test" data-isBool="true" data-someNumber="123"></div>

Does work 🙂

<div id="test" data-is-bool="true" data-some-number="123"></div>

‘.stop()’ Collaborate & Listen

Binding jQuery animations to mouse events is a key part of modern web-based user interaction. It’s also something that you see done poorly on even the most famous of web sites. This article provides a straight forward example of built up animations and demonstrates how visually jarring they can be. Thankfully it’s easily fixed with a single function prefix or a parameter added to $.animate calls.

When using $.animatequeue: false can be added to the parameters to prevent chaining. Animation shortcuts such as$.fadeIn or $.slideDown do not take queue settings. Instead you have to pre-empt these animations with the $.stop method of pausing currently executing animations. Certain scenarios require the animation to stop dead in its tracks, or to jump to the end of the transition. It is recommended you familiarise yourself with the documentation of the parameters clearQueue andjumpToEnd, because god knows I can’t help you there.

$("selector").stop(true,true).fadeOut();

$("selector").animate({
	property: value
}, {
	duration: 1000,
	queue: false
}

Optimise Your Selectors

jQuery is pretty chill. It can do pretty much everything but make you coffee, and I hear that’s in the roadmap for 2.0. One thing you have to be careful about is abusing the power that is the sizzleJS selector engine. There are two strategies to overcome this: caching the selector results and using efficient selectors.

CACHING SELECTOR RESULTS

Do a costly DOM query every time you want to change something, or store a reference to the element? Pretty clear choice.

// before
$(".quote a").bind("click", doStuff); // DOM query eww

// now
$(".quote a").addClass("quoteLink"); // DOM query eww

// later
$(".quote a").fadeIn("slow"); // DOM query eww

Ignoring chaining, this is better:

// before
var $quoteLinks = $(".quote a");  // the only DOM query
$quoteLinks.bind("click", doStuff);

// now
$quoteLinks.addClass("quoteLink");

// later
$quoteLinks.fadeIn("slow");

USING EFFICIENT SELECTORS

So jQuery/sizzleJS can use CSS3 selectors like a boss, but what’s the real cost? Behind the scenes the browser is hopefully using document.querySelector(), but there’s also a fair chance it will be breaking down your selector string and querying the DOM manually.

// an ID search is the quickest possible query, then it just takes a list of the childNodes and matches the class
$("#quoteList").children(".quotes"); 

// looks for the "foo" class only in the pre-defined bar element
$(".foo",bar);

A ‘for’ Loop is Always Quicker Than a ‘each()’ Loop

No matter what happens in the next few years of browser development, a native for loop will always be quicker than a jQuery$.each() loop. When you think of what jQuery really is (a library wrapped around native JS functions) you begin to realise that the native underlying JavaScript is always going to be quicker. It’s a tradeoff of run speed versus authoring speed.

It is vital that a native for loop is always used for performance critical functions that could fire potentially hundreds of times per second. Examples include:

  • Mouse movement
  • Timer intervals
  • Loops within loops

CSS

Understanding the Box Model is Key

The “box model” is a key determining factor in how a browser renders your page. A healthy understanding of it’s intricacies will make your job so indescribably easier. The box model denotes the way in which the physical dimensions of a HTML element are calculated. If a block element has a fixed width of say, 100px, then how should the padding, border and margin be placed?

Plenty of websites offer in depth descriptions, but put simply: the standards compliant implementation places the border and padding outside of the specified width. It’s best explained with a graphic. Consider this code:

/* the old way (178 + 20 + 2 = 200) */
.foo {
	width: 150px;
	height: 150px;
	padding: 25px;
	border: 25px solid;
	margin: 20px;
}

WHAT YOU WOULD EXPECT (QUIRKS MODE)

The padding and border are calucated inward, preserving the height and width specifically set to be 150px.

Box Model - Quirks Mode

WHAT YOU GET (STANDARDS COMPLIANT MODE)

Instead, you get 250px. 150px + (2 * 25) + (2 * 25).

Box Model - Standards Mode

If you think it seems odd, you’re not alone. There is a fix at hand, and it involves a CSS property called box-sizing, and it works in IE8 and above. It allows you to choose the exact way in which an elements dimensions are calculated, and its a lifesaver. Parameter support varies and vendor prefixes apply, so consult caniuse for specifics.

/* the old way (178 + 20 + 2 = 200) */
.foo {
	width: 178px;
	padding: 10px;
	border: 1px;
}

/* a better way */
.foo {
	width: 200px;
	padding: 10px;
	border: 1px;
		-webkit-box-sizing: border-box;
		-moz-box-sizing: border-box;
		box-sizing: border-box;
}

While it was always possible to mentally calculate widths by removing pixel units from each other (as per the first method), it was never entirely clear how to do so with variable width units like percentages and EMs. There was no other solution at this point besides wrapping elements in parent elements to ensure widths and padding/margin/borders could all be separate.

Know when to Float, and when to Position

Gone are the days of table based layouts. The moment we admit that we can concentrate our efforts into better understanding the way floats and positions work. There’s a particular mental model that needs to be grasped, and I believe this is something best done with practise.

Floats are great for sucking elements out of the DOM and forcing them hard up against a left or a right edge. They became the bread and butter of the post table layout stage in front end dev, possibly because of the poor browser support of display: inline and inline-block, as well as z-index bugs stemming from position support. These days there really is no excuse. Inline-block is fairly well supported, and a quick hack will get it working in IE7.

The arguments that previously held back absolutely positioning elements with CSS have thankfully died down. In theory, positioning allows you to place elements on a page (or within any container for that matter) with Xs and Ys in a straightforward manner that should be familiar to people like Flash developers.

UNDERSTANDING POSITIONS

It’s important to understand one fact when positioning elements with CSS: the position is always relative to the nearest positioned parent element. When people first start dabbling with CSS, there’s a common misconception that position: absolute; positions right up to the page root. I think this stems from the fact that, yes, without any parent elements with position styles – this is true. It traverses up the DOM tree, not finding any positioned elements, and settles on the root of the page.

So if position: absolute; pulls elements out of their normal flow, how do you position an element relative to it’s parent? That’s straight forward. The parent element needs to by styled position: relative;, and then all child elements will draw from the top, right, bottom and left of this parent container. Using this knowledge, how would you go about achieving the following straightforward layout?

How would go about coding up this image?

Using float, you would need to wrap the items in a clearfix, float .one left, and fiddle with floats and margins on both .twoand .three. You would end up with something similar to the following:

.parent {
	/* ghetto clearfix */
	width: 310px;
	overflow: auto;
}
.one {
	width: 200px;
	height: 210px;
	float: left;
}
.two {
	width: 100px;
	height: 100px;
	float: right;
	margin-bottom: 10px;
}
.three {
	width: 100px;
	height: 100px;
	float: right;
}

Using position allows us to, as described earlier, paint the elements on the screen with x and y co-ordinates in a very explicit way. While the above method with floats will break long lines down the page, the below method will keep everything in place, regardless of content.

.parent {
	position: relative;
	width: 310px;
	height: 210px;
}
.one, .two, .three {
	position: absolute;
}
.one {
	top: 0;
	left: 0;
	width: 200px;
	height: 210px;
}
.two {
	top: 0;
	right: 0;
	width: 100px;
	height: 100px;
}
.three {
	bottom: 0;
	right: 0;
	width: 100px;
	height: 100px;
}

As mentioned earlier, there are z-index issues to be considered. While the above example might seem a bit excessive, once you start thinking with positions, it will opens a world of possibilities.

Whitespacing

Whitespacing of CSS can be difficult as we chop and change between single and multi line CSS arguments. I’m not going to get into that.

PROPER SPACING

/* BAD */
.selector {display:none;background:#ff0000;color:#000000;} 

/* GOOD - SINGLE LINE */
.selector { display: none; background: #ff0000; color: #000000; } 

/* GOOD - MULTI-LINE */
.selector {
	display: none;
	background: #ff0000;
	color: #000000;
}

SAME LINE BRACES

.selector {
	display: none;
	background: #ff0000;
	color: #000000;
}

INDENTING CHILD ELEMENTS

Purely optional, and personally only employed when in a document with single line declarations.

.selector { display: none; background: #ff0000; color: #000000; }
	.selector a { text-decoration: none; }
	.selector span { font-weight: bold; }

GROUPING & INDENTING VENDOR PREFIXES

.selector {
	background: #FFF; border: 1px solid #000; color: #EAEAEA;
		-webkit-border-radius: 3px;
		-moz-border-radius: 3px;
		border-radius: 3px;
}

CSS Shorthand

GROUPING PROPERTIES

Grouping properties together is one of the single most effective methods to greatly reduce the size of a CSS file. It’s important to understand how properties are ordered (clockwise – top, right, bottom, left) and how they can be further shortened (top and bottom, left and right).

/* LONG CODE IS LONG */
padding-top: 1px;
padding-right: 2px;
padding-bottom: 1px;
padding-left: 2px;

/* BETTER */
padding: 1px 2px 1px 2px;

/* BEST */
padding: 1px 2px;

FROM 0PX TO HERO

Assigning a unit type to a property value of zero is redundant. It is not important to know whether an element should be 0pxfrom the left or 0 elephants from the left, just that it’s bang on the left.

/* BAD */
padding: 0px 10px;
/* GOOD */
padding: 0 10px;

Commenting Blocks

Commenting large blocks of CSS is a great way of keeping track of multiple style areas within the one stylesheet. Obviously it works better with single line CSS styles, but the effect is not entirely lost on multi-line either. The use of dashes versus equals versus underscores are all up the individual, but this is how I like to manage my stylesheets.

/* === HORIZONTAL NAV === */
#horizNav { width: 100%; display: block; }
#horizNav li { display: block; float: left; position: relative; }
#horizNav li a { display: block; height: 30px; text-decoration: none; }
#horizNav li ul { display: none; position: absolute; top: 30; left: 0; }

/* === HOME PAGE - CAROUSEL === */
#carousel { width: 960px; height: 150px; position: relative; }
#carousel img { display: none; }
#carousel .buttons { position: absolute; right: 10px; bottom: 10px; }

Clearing Floats

Clearing a <div> used to mean extra DOM, because it involved adding an extra clearer element. The better way is to set a specific width on the parent element (“auto” doesn’t work in all browsers and scenarios) and an overflow value of either “auto” or “hidden”. “Hidden” obviously degrades better, but there are some IE compatibility versions where “auto” works better.

THE HTML:

<div class="parentElement">
	<div class="childElement">
		I'm floated left!
	</div>
	I'm normal text that wraps around the float
</div>

THE CSS:

.parentElement {
	width: 100%;
	overflow: hidden;
}
.childElement {
	float: left;
}

Contributors have also brought the latest clearfix to my attention. The micro clear-fix is considered stable and cross browser compliant enough to make it to the latest HTML5 boiler plate release. I highly recommend you check it out. Although I am not a massive fan of browser-specific CSS and pseudo elements such as :after, the micro clearfix is definitely more robust. It also prevents top margins from collapsing which is an absolute life saver.

Vertical & Horizontal Centering

Centering elements horizontally is not exactly rocket science, and I’m sure most of you are familiar with the following snippet:

.class {
	width: 960px;
	margin: 0 auto;
}

Front end devs have been using this snippet for a long time, without fully understanding why it didn’t work vertically. From my understanding, it’s important to remember that the parent element will generally have a height: auto; on it, and that there is no 100% height in which to vertically center the element. Applying the position: absolute; effectively moves the element out into position mode and responds to the pushing and pulling of auto margins and no specific location.

.exactMiddle {
	width: 100px;
	height: 100px;
	position: absolute;
	top: 0;
	right: 0;
	bottom: 0;
	left: 0;
	margin: auto;
}

The downsides of this method include its lack of support in IE6 and IE7, and the absence of a scroll bar if the browser is resized to be smaller than the centered object. There are more methods on this page (this is Method 4), but this is by far the best.

The vertical centering of text in an element is also straightforward. If the text is on a single line, like a horizontal navigation item, you can set the line-height to be that of the element’s physical height.

#horizNav li {
	height: 32px;
	line-height: 32px;
}

Feature Sniff, Don’t Browser Sniff

In the earlier discusison of JavaScript feature detection, applying properties if a browser is any version of IE is increasingly problematic. Man-of-steel Paul Irish pioneered the use of IE version sniffing to address this problem, but Modernizr has since come to the rescue. Modernizr places classes on the root <html> element specifying whether features are supported. Bleeding edge styles can then easily cascade from (or be removed from) these classes.

.my_elem {
   -webkit-box-shadow: 0 1px 2px rgba(0,0,0,0.25);
   -moz-box-shadow: 0 1px 2px rgba(0,0,0,0.25);
   box-shadow: 0 1px 2px rgba(0,0,0,0.25);
}

/* when box shadow isn't supported, use borders instead */
.no-boxshadow .my_elem {
   border: 1px solid #666;
   border-bottom-width: 2px;
}

You’re Not !important

A reliance upon the !important tag is a dangerous thing. The cases that warrant its use are rare and specific. They revolve around the necessity to override another stylesheet which you do not have access or permission to edit. Another scenario is hard coding an element’s styles to prevent inline JavaScript styles from taking precedence. Instead !important is used as a lazy shortcut to set the priority of your style over another, causing headaches further down the line.

The use of the !important tag can be mostly avoided via the better understanding of CSS selector precedence, and how to better target elements. The more specific the selector, the more likely it will be accepted as the applicable style. The following example from vanseodesign demonstrates the specificity at work.

p { font-size: 12px; }
p.bio { font-size: 14px; }

Their article on style precedence does a better job explaining inheritence than I ever could, so please give it a go.

Aggressive Degradation

It’s worth noting that this is a personal opinion, and best suited to very specific situations. The stance of aggressive degradation will not be well received in large commercial projects or enterprise solutions relying upon older browsers.

Aggressive degradation dictates that if a particular (older) browser cannot render a certain effect, it should simply be omitted. A CSS3 button is a good example. Effects such as border-radiusbox-shadowtext-shadow and gradients will be displayed in cutting edge browsers. A graceful fallback of a .PNG would be provided for slightly older browsers, and the most graceful of all solutions would include a PNG-Fix for IE6 or the use of filter arguments to replicate gradients and shadows. However, aggressive degradation in this situation instructs you to neglect the older browsers and present them with a flat, satisfactory object.

Put simply, aggressive degradation boils down to: if your browser can’t render a gradient or a box shadow, tough luck.

While not ideal for every situation, it ensures the timely delivery of projects and that the root product is still usable and not reliant on (validation breaking) hacks.

CSS3 & HTML5

Feature Sniff with Modernizr

I think I’ve gone on enough about this already. Use Modernizr to detect the availability of specific HTML5 and CSS3 features.

@font-face Use and Abuse

Before you consider embedding a custom font, is important that you inspect the EULA and check if web embedding is allowed. Foundries are understandably reluctant to allow designers and developers the ability to place font files directly on a server which can then be copied by a savvy end user. Particular foundries also prohibit the embedding of particular file types, such as.TTF and .OTF.

If, after careful consideration, you believe the desired font is web embeddable: head on over to the Font Squirrel @font-face Generator. It utilises Fontspring’s bulletproof @font-face structure and automatically generates all the required file formats.

Degradation

Thankfully browser handling of unsupported HTML5 and CSS3 is already that of a graceful nature. New additions to the list of<input /> types such as “email”, “search” etc. will generally degrade to normal <input type="text" /> when not natively supported. Similarly, CSS3 properties that aren’t supported will simply not appear. Responsive layouts controlled by height and width media queries are simply not applied.

Subtle CSS3 effects should be employed as a reward for users who run a modern browser.

The resources section below includes a few libraries to help normalise HTML5 and CSS3 functionality across a range of older browsers.

Resources

Useful Resources

The following resources are vital for the standardisation of code and interaction in a modern web page. They ensure that CSS3 and HTML5 features are made accessible across a range of browsers that previously lacked support.

Support and Suggestions

This document was prepared by Tait Brown (@taitems).

Questions, corrections and suggestions can be lodged on the GitHub repository for this page. You can also fork your own branch and add your own company/product specific guidelines.