Friday, June 24. 2016
Integrating runalyze into runnerup (Part I)
As Marx2 told me in the github pull request commented in the previous entry, the runalyze project is really nice. The previous weekend I was testing and uploading some of my running activities just to try it. And, in general, the impression is quite good. Obviously is much more complete than my little application RunnerUpWeb and works smoothly (actually the application is much more than what I needed and looked for).
At the same time I studied how the files were added (just in case I have the time to integrate it inside runnerup phone application). I tested with TCX files (the ones I usually use in my application and, therefore, I had easy access to them) and the process that runalyze uses is, at least, strange. When you want to upload any type of file the procedure that is followed is more or less the following:
When the upload button is clicked you can choose a file from your file system (a lot of formats are allowed, TCX among them).
The file is uploaded using a multipart/form-data post but nothing more, I mean, the file is uploaded to a directory inside the server but no real parsing is done. The file is just placed in the system (in a folder of the system more concretely).
After the file is uploaded another request to the server triggers the parsing (TCX parsing in my case). And the result of this operation is creating a huge HTML form with a lot of inputs (the real data obtained from the TCX file but exposed as a form). The HTML form is incrusted in the page as another tab in the import screen.
In that form the user can modify some inputs of the activity (some information that can be added or modified to the parsed data).
Finally the user submits that form and the activity is actually saved in the system (inside the ddbb).
At first I was thinking about replicating the same procedure with runnerup but it is a waste of time. I think now that the most proper way of integrating runnalize into a runnerup is just constructing the post data directly (all the needed input that the import process generates from the TCX file in the web application) and sending that to the runalyze. This idea means that the phone should generate a proper form data to post and only executes the last point of the procedure (point 5 in the previous explanation). Obviously that information should be generated reading the activity data stored in the phone database (exactly as it is done for generating a TCX or GPX file).
I am going to add here the java method that was tested to send the data (raw data copied from the browser) to the runalyze server.
public void submit() throws Exception {
URL url = new URL(url + "/call/call.Training.create.php?json=true");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setRequestMethod("POST");
conn.setInstanceFollowRedirects(false);
conn.setDoOutput(true);
addHeadders(conn); // set the session cookie into the request
try (OutputStreamWriter writer = new OutputStreamWriter(conn.getOutputStream())) {
StringBuilder sb = new StringBuilder()
.append("creator=").append("&")
.append("creator_details=").append("&")
.append("activity_id=1461954855").append("&")
.append("timezone_offset=120").append("&")
.append("arr_time=2|4|6|...").append("&")
.append("arr_lat=40.463737|40.463726|40.46372|...").append("&")
.append("arr_lon=-3.7070801|-3.7070918|-3.707106|...").append("&")
.append("arr_geohashes=").append("&")
.append("arr_alt=754|753|753|...").append("&")
.append("arr_alt_original=").append("&")
.append("arr_heart=78|80|80|...").append("&")
.append("arr_dist=0|0.00161|0.00309|...").append("&")
.append("arr_cadence=").append("&")
.append("arr_power=").append("&")
.append("arr_temperature=").append("&")
.append("arr_groundcontact=").append("&")
.append("arr_vertical_oscillation=").append("&")
.append("arr_groundcontact_balance=").append("&")
.append("pauses=[{\"time\":302,\"duration\":2,\"hr-startv\":156,\"hr-end\":156},{\"time\":570,\"duration\":2,\"hr-start\":0,\"hr-end\":0},{\"time\":848,\"duration\":2,\"hr-start\":0,\"hr-end\":0},{\"time\":1138,\"duration\":2,\"hr-start\":0,\"hr-end\":0},{\"time\":1306,\"duration\":4,\"hr-start\":0,\"hr-end\":0},{\"time\":1448,\"duration\":2,\"hr-start\":0,\"hr-end\":0},{\"time\":1722,\"duration\":2,\"hr-start\":0,\"hr-end\":0},{\"time\":1908,\"duration\":2,\"hr-start\":0,\"hr-end\":0},{\"time\":1912,\"duration\":2,\"hr-start\":0,\"hr-end\":0},{\"time\":2038,\"duration\":2,\"hr-start\":0,\"hr-end\":0}]").append("&")
.append("hrv=").append("&")
.append("fit_vdot_estimate=0.00").append("&")
.append("fit_recovery_time=0").append("&")
.append("fit_hrv_analysis=0").append("&")
.append("fit_training_effect=").append("&")
.append("fit_performance_condition=").append("&")
.append("elapsed_time=2288").append("&")
.append("elevation_calculated=0").append("&")
.append("groundcontact=0").append("&")
.append("vertical_oscillation=0").append("&")
.append("groundcontact_balance=0").append("&")
.append("vertical_ratio=0").append("&")
.append("stroke=").append("&")
.append("stroketype=").append("&")
.append("total_strokes=0").append("&")
.append("swolf=0").append("&")
.append("pool_length=0").append("&")
.append("weather_source=").append("&")
.append("is_night=0").append("&")
.append("distance-to-km-factor=1").append("&")
.append("sportid=1").append("&")
.append("typeid=1").append("&")
.append("is_race_sent=true").append("&")
.append("time_day=29.04.2016").append("&")
.append("time_daytime=18:34").append("&")
.append("s=37:46").append("&")
.append("kcal=554").append("&")
.append("pulse_avg=152").append("&")
.append("pulse_max=174").append("&")
.append("distance=7.851").append("&")
.append("elevation=0").append("&")
.append("pace=4:49").append("&")
.append("power=0").append("&")
.append("cadence=0").append("&")
.append("splits[km][]=1.006").append("&")
.append("splits[time][]=5:00").append("&")
.append("splits[active][]=1").append("&")
.append("splits[km][]=1.005").append("&")
.append("splits[time][]=4:30").append("&")
.append("splits[active][]=1").append("&")
.append("splits[km][]=1.004").append("&")
.append("splits[time][]=4:40").append("&")
.append("splits[active][]=1").append("&")
.append("splits[km][]=1.000").append("&")
.append("splits[time][]=4:52").append("&")
.append("splits[active][]=1").append("&")
.append("splits[km][]=1.004").append("&")
.append("splits[time][]=5:16").append("&")
.append("splits[active][]=1").append("&")
.append("splits[km][]=1.003").append("&")
.append("splits[time][]=4:36").append("&")
.append("splits[active][]=1").append("&")
.append("splits[km][]=1.004").append("&")
.append("splits[time][]=5:22").append("&")
.append("splits[active][]=1").append("&")
.append("splits[km][]=0.875").append("&")
.append("splits[time][]=3:50").append("&")
.append("splits[active][]=1").append("&")
.append("use_vdot=on").append("&")
.append("rpe=").append("&")
.append("comment=").append("&")
.append("partner=").append("&")
.append("route=").append("&")
.append("notes=").append("&")
.append("weatherid=1").append("&")
//.append("temperature=").append("&")
.append("wind_speed=").append("&")
.append("wind_deg=").append("&")
.append("humidity=").append("&")
.append("pressure=").append("&")
.append("tag_old=").append("&")
.append("equipment_old=").append("&")
.append("submit=submit");
writer.write(sb.toString());
writer.flush();
conn.connect();
String response = getResponse(conn.getInputStream());
if (conn.getResponseCode() != 200 || !response.contains("The activity has been successfully created.")) {
throw new IllegalStateException("Error code " + conn.getResponseCode());
}
}
}
The java class uses a technique to save the headers of a previous login (as runnerup is doing right now) and add them to the next submit. That login method is not shown in the entry. As you see the activity data is sent using a lot of inputs. Some of them are arrays of numbers separated by a pipe (all the data for samples: time, latitude, longitude, altitude, distance,...). Others are just some fixed data (as the day and time of the activity, the pace, the sport and type,...). For example the information about pauses is more complicated (it is a json with time, duration and heart-rate at the beginning and the end of each pause). One input line is commented out (temperature), just to check if you can omit optional fields. If you sent all that information a new activity is created in runalyze. So now the problem is obtaining all those inputs from the internal runnerup database for the activity to export. My main concern about the process is how much stable it will be. Will runalyze change the way activities are created? Is this form more or less stable? If someone involved with the project reads the entry please comment below or contact me directly.
In general now I am quite busy and I do not know if I will have the time to develop all the synchronizer stuff. I am just creating this entry to not forget what I have already thought and to encourage any of you to also start the task. The problem seems to be just reading from the database and filling the post data (but you know that it will never be so easy at the end). I will try to develop the synchronizer but I cannot say when I will do it. So, if you have the time, please go for it.
Regards!
Saturday, June 11. 2016
Making the changes for RunnerUpWeb
If you think that the RunnerUpWeb project was forgotten or stopped you could not be more wrong. I have been busy the previous weeks trying to integrate the backend as a regular account in the android application. As a result now there are two pull requests waiting in the github repository:
A change to make that the runnerup application generates valid TCX files (compatible with the associated XSD). This way the application ensures that the file, uploaded via phone or browser, is valid and not another (maybe dangerous) thing.
The last pull request adds a new account that represents your own RunnerUpWeb installation. This is the main enhancement to make the phone app understand a RunnerUpWeb backend.
Obviously the second point is the important one. Integrating the account into the phone counterpart is the main step to make this project start up (anyone would be able to configure easily his RunnerUpWeb account in the phone). Once this point was overcome I can continue developing the backend application in order to release a first (more or less) stable version.
Adding the account was much more difficult than expected because normal backends in runnerup have a fixed URL to connect to (Digifit, endomondo, sports-tracker and so on always connect to the same hostname for storing the activities). Nevertheless RunnerUpWeb is a FOSS application you can install wherever you like (your home PC, your hosting web server,...), so the URL the app should connect to is variable and not fixed like in all the other accounts. For the moment I have implemented an easy workaround. Until now each account can store an authentication configuration (username and password), with my request there is another type of authentication that adds the URL of the service to the previous information (username, password and URL are now saved in the auth config). With that change I could extend the former DigifitSynchronizer class with some minor changes to make it compatible with my application.
A second nasty issue (but this one I think it is caused because my complete ignorance programming in android) was that the image for my account were reassigned to other accounts. Finally I understood that the application was assigning the images in (lexicographic) order and my a16 image was before a2, a3 and so on. I renamed my logo to b01_runnerupweb.png and I think everything works as expected now. But I suppose that people from runnerup project will recommend me something better for this problem.
I present a little video using the android studio to configure and connect a virtual device to my development environment. First the modified application is deployed to the phone. Then I show that the new account is there ready to be configured. A new user is created in the backend server. The account is created against the demo server and some activities are uploaded. Finally the user logs in the backend server and the activities are there. Really nice, isn't it?
Some hours after my pull request was posted, Marx2 commented about Runalize. It seems that it is another try to make another free source backend server for sport activities. I simply did not find it when I looked for that type of application. Let's see what happens because Runalize has no account in runnerup neither. My personal opinion is that they should have been trying what I am doing long time ago. That way they have saved me a lot of time and effort. For the moment I will keep pushing to integrate my changes in runnerup. I suppose that Runalize also needs URL configuration for the account. So the change can help both.
Enjoy!
Tuesday, June 7. 2016
WBXML stream: Version 0.2.0
Some days ago pwnslinger opened the first issue in the wbxml-stream project. If you remember that project is my little effort to provide a wbxml parser/encoder for java/StAX. Mainly the problem was that the library did not parse correctly WBXML documents in version 1.1. The WBXML standard has four versions, from 1.0 to 1.3, and there are subtle differences between them. The version 0.1.0 of the wbxml-stream library just managed version 1.3 (only 1.3 documents could be encoded and the previous versions were just parsed like if they were the last one, no differences were taken into account). Nevertheless version 1.1 had an issue in the defined enumeration (a copy/paste problem :-/ ) and it was not recognized by the library.
That issue made me improve the implementation in order to properly manage the four different versions of the specification. This way version 0.2.0 has been released with the feature of handling better with previous versions of the standard. Now it is possible to encode a WBXML document in any version. Besides the parsing/encoding of documents take into account the specific characteristics between versions. The first version 1.0 does not add the charset of the document (it just manages the unknown encoding) and it does not recognize opaques. Version 1.1 adds tag opaques and charset/encodings but attribute opaques are added later in version 1.2. Version 1.2 also adds page switches to increase the number of tags, attributes and values a definition can handle. There is one difference between version 1.0 and 1.1 which wbxml-stream does not manage. In version 1.1 the document body is defined as *pi element *pi (an element with optional processing instructions before and after it), in contrast, in version 1.0 the body was 1*content (one or more content, which in turn can be an element, a string, a extension, an entity or a processing instruction). This previous definition is quite weird, a WBXML document can be only a string or an entity, which is clearly not a valid XML document. For that reason I decided to forget about this difference (a WBXML document version 1.0 with that strange content will not be correctly parsed, throwing an exception for sure). A page in the project wiki summarizes those differences if you are interested in the details.
The new version of the library also adds some improvements in the management of encoding (now the default charset, encoding for unknown, is UTF-8 instead of ASCII) and numeric character references (things like ñ or ñ to reference a character ñ). In the latter I am not sure if everything is right and maybe a new version will be needed but, for sure, it is in a better condition than in the previous version. The WbXmlOutputFactory has been updated in order to receive the version we want to write the WBXML with (property es.rickyepoderi.wbxml.stream.version). Besides the command Xml2WbXml now admits two more options (-c or --charsert and -v or --version) to convert the XML file to WBXML using the encoding and the version specified.
Here it is a little snippet to use the new version 0.2.0 to convert an SL xml file into WBXML v1.1 using a specific encoding.
// read the XML using DOM
InputStream in = new FileInputStream("sl-001.xml");
DocumentBuilderFactory domFact = DocumentBuilderFactory.newInstance();
domFact.setNamespaceAware(true);
domFact.setIgnoringElementContentWhitespace(true);
DocumentBuilder domBuilder = domFact.newDocumentBuilder();
Document doc = domBuilder.parse(in);
Element element = doc.getDocumentElement();
// locate the definition of the WBXML using the name
WbXmlDefinition definition = WbXmlInitialization.getDefinitionByName("SL 1.0");
// create the StAX stream writer using the definition
OutputStream out = new FileOutputStream("sl-001.wbxml");
XMLOutputFactory fact = new WbXmlOutputFactory();
fact.setProperty(WbXmlOutputFactory.DEFINITION_PROPERTY, definition);
fact.setProperty(WbXmlOutputFactory.VERSION_PROPERTY, WbXmlVersion.VERSION_1_1);
XMLStreamWriter xmlStreamWriter = fact.createXMLStreamWriter(out, "ISO-8859-1");
// create a transformer to convert DOM into StAX
Transformer xformer = TransformerFactory.newInstance().newTransformer();
Source domSource = new DOMSource(doc);
StAXResult staxResult = new StAXResult(xmlStreamWriter);
xformer.transform(domSource, staxResult);
And here I present a execution of the command trying to convert a SI XML file into WBXML version 1.0 and encoding ISO-8859-1 (remember that v1.0 does not use encoding, therefore any character outside ascii is compromised). As the SI document uses some opaques (which are not defined in version 1.0) the implementation avoids the use of the opaque and some warning messages are displayed (in general issues with versions generate warnings or throw exceptions, in this case the encoding does not use the opaque and warns the user because the resulting document probably is invalid).
$ java -cp wbxml-stream-0.2.0.jar es.rickyepoderi.wbxml.tools.Xml2WbXml -d "SI 1.0" -v 1.0 -c ISO-8859-1 si.xml si.wbxml Jun 06, 2016 7:23:59 PM es.rickyepoderi.wbxml.document.WbXmlEncoder encode WARNING: Opaque not used for attribute "created" in element "indication" because version "1.0" does not accept attribute opaques. Jun 06, 2016 7:23:59 PM es.rickyepoderi.wbxml.document.WbXmlEncoder encode WARNING: Opaque not used for attribute "si-expires" in element "indication" because version "1.0" does not accept attribute opaques.
And that is all. If you are using wbxml-stream library for something please try to use the new version because it integrates a new nice feature (version management) and some minor improvements and bug fixes. Stay connected for more news about the project.
Cheerio!
Comments