[mlpack] Augmented RNNs @ GSoC'17

Marcus Edel marcus.edel at fu-berlin.de
Tue Mar 7 09:54:32 EST 2017


Hello Konstantin,

thanks for getting back.

> Since we have to make several different augmented RNN models, I think it is a
> good idea to make a namespace mlpacK::ann::augmented and a class BaseAugmented
> (will be useful for benchmarking later).

Putting different networks under a unified namespace is a good idea. Regarding
the BaseAugmented class, we should avoid inheritance (or, at least, virtual
inheritance) in deference to templates. This is because virtual functions incur
runtime overhead. In this case I don't see a compelling reason to introduce a
BaseAugmented class, maybe I missed something? We could do:

Code:
=====

class CopyTask
{
  template<typename ModelType>
  void Evaluate(ModelType& model)
  {
    // Call model.Train() using the generated copy task data.
    // Call model.Evaluate
  }
}

Usage:
======

NTM ntm;
CopyTask.Evaluate(ntm);

What do you think about the design?

> P.S. Some questions that arose while trying to get in grips with mlpack:
> - What resources can you advice to brush up *advanced/modern* C++? (e.g., templates, && in functions)

There are some really nice books that might help to refresh your knowledge:

- "Modern C++ Design, Generic Programming and Design Patterns Applied" by Andrei Alexandrescu
- "Effective C++" by Scott Meyers
- "Effective STL" by Scott Meyers

There are also some references on http://mlpack.org/gsoc.html

> - Can you show me the working example of mlpack::ann::FFN compatible with
> current upstream C++ API?

Not sure what you mean, can you elaborate more?

I hope this is helpful, let us know if you have any more questions.

Thanks,
Marcus

> On 5 Mar 2017, at 10:12, Сидоров Константин <kost2906 at yandex.ru> wrote:
> 
> Hello Marcus!
> Right now, I'm thinking quite a lot on "Augmented RNNs" project. For example, I'm trying to convert ideas from project description to something more concrete and mlpack-specific. Here is my first shot at C++ API for NTM.
> Disclaimer: I haven't written on C++ for ~10 months. My last experience of C++ coding is our (Russian) national programming olympiad (April '16), after which I've been coding almost exclusively on Python.
> Since we have to make several different augmented RNN models, I think it is a good idea to make a namespace mlpacK::ann::augmented and a class BaseAugmented (will be useful for benchmarking later).
> Inside it, we can define class NTM< OutputLayerType, InitializationRuleType > : BaseAugmented with standard mlpack interface:
> - template<typename NetworkType, typename Lambda> NTM(NetworkType controller, arma::mat &memory, Lambda similarity)
> - Predict (arma::mat &predictors, arma::mat &responses)
> - Train (const arma::mat &predictors, const arma::mat &responses)
> For benchmarking, we can define class mlpack::ann:augmented::BaseBenchmark with interface:
> - BaseBenchmark() = 0
> - Evaluate() = 0 (the method that will accept task parameters as arguments and run the augmented model)
> As an example, an API of CopyBenchmark : BaseBenchmark:
> - CopyBenchmark()
> - Evaluate(BaseAugmented model, int maxLength = 5, int repeats = 1) // repeats is a parameter to convert copy to repeat-copy task.
> So, that's some kind of API. I would be interested to discuss and analyze it with you.
>  
> P.S. Some questions that arose while trying to get in grips with mlpack:
> - What resources can you advice to brush up *advanced/modern* C++? (e.g., templates, && in functions)
> - Can you show me the working example of mlpack::ann::FFN compatible with current upstream C++ API?
>  
> 28.02.2017, 16:43, "Marcus Edel" <marcus.edel at fu-berlin.de <mailto:marcus.edel at fu-berlin.de>>:
>> Hello Konstantin,
>>  
>>  
>>> My name is Konstantin Sidorov, and I am an undergraduate student in Astrakhan
>>> State University (Russia). I’m glad to know that mlpack was accepted in GSoC’17
>>> – as a side note, congratulations :)
>>  
>> thanks and welcome!
>>  
>>  
>>> I’m already fairly familiar with deep learning. For example, recently I
>>> implemented optimality tightening from “Learning to play in a day”
>>> (https://arxiv.org/abs/1611.01606 <https://arxiv.org/abs/1611.01606>) for the AgentNet (“Deep Reinforcement
>>> Learning library for humans”, https://github.com/yandexdataschool/AgentNet <https://github.com/yandexdataschool/AgentNet>).
>>  
>> Sounds really interesting, the "Learning to Play in a Day" paper is on my
>> reading list, looks like I should move it up.
>>  
>>  
>>> Of course, on such an early stage I have no detailed plan what (and how) to do –
>>> only some ideas. In the beginning, for example, I’m planning to implement NTMs
>>> as described in arXiv paper and implement *reusable* benchmarking code (e.g.,
>>> copy, repeat copy, n-grams). I would like to discuss this project more
>>> thoroughly if possible. In addition, this is my first participation in GSoC. So,
>>> excuse me in advance if I’ve done something inappropriate.
>>  
>> Implementing the NTM task from the paper, so that they can be used for other
>> models as well is a great idea. In fact, you see a lot of other papers that at
>> least reuse the copy task. There are a bunch of other interesting tasks that
>> could be implemented like the MNIST pen stroke classification task recently
>> introduced by Edwin D. de Jong in his "Incremental Sequence Learning" paper. The
>> Stanford Natural Language Inference task proposed by Samuel R. Bowman et al. in
>> "A large annotated corpus for learning natural language inference" can be also
>> transformed into a long-term dependency task, that might be interesting.
>>  
>> Regarding the project itself, take a look at other models as well, depending on
>> the model you choose, I think there is some time left for another model. Also,
>> about the implementation, mlpack's architecture is kinda different to Theano's
>> graph construction and compilation work, but if you managed to work with Theano
>> you shouldn't have a problem.
>>  
>> If you like we can discuss any details over the mailing list and brainstorm some
>> ideas, discuss an initial class design, etc.
>>  
>> I hope this is helpful, let us know if you have any more questions.
>>  
>> Thanks,
>> Marcus
>>  
>>> On 28 Feb 2017, at 07:06, Сидоров Константин <kost2906 at yandex.ru <mailto:kost2906 at yandex.ru>> wrote:
>>>  
>>> Hello Marcus,
>>> My name is Konstantin Sidorov, and I am an undergraduate student in Astrakhan State University (Russia). I’m glad to know that mlpack was accepted in GSoC’17 – as a side note, congratulations :)
>>> I’m interested to work on project “Augmented Recurrent Neural Networks”. I’m already fairly familiar with deep learning. For example, recently I implemented optimality tightening from “Learning to play in a day” (https://arxiv.org/abs/1611.01606 <https://arxiv.org/abs/1611.01606>) for the AgentNet (“Deep Reinforcement Learning library for humans”, https://github.com/yandexdataschool/AgentNet <https://github.com/yandexdataschool/AgentNet>). Here is the merged pull request: https://github.com/yandexdataschool/AgentNet/pull/88 <https://github.com/yandexdataschool/AgentNet/pull/88>.
>>> As you see, I’m quite familiar with deep learning and Theano. Even though my main field of interest is RL, I would be very interested in doing something new – that is why I’ve chosen “Augmented RNNs”.
>>> Of course, on such an early stage I have no detailed plan what (and how) to do – only some ideas. In the beginning, for example, I’m planning to implement NTMs as described in arXiv paper and implement *reusable* benchmarking code (e.g., copy, repeat copy, n-grams).
>>> I would like to discuss this project more thoroughly if possible. In addition, this is my first participation in GSoC. So, excuse me in advance if I’ve done something inappropriate.
>>> ---
>>> Best Regards,
>>> Konstantin.
>>> _______________________________________________
>>> mlpack mailing list
>>> mlpack at lists.mlpack.org <mailto:mlpack at lists.mlpack.org>
>>> http://knife.lugatgt.org/cgi-bin/mailman/listinfo/mlpack <http://knife.lugatgt.org/cgi-bin/mailman/listinfo/mlpack> 
>  
> --
> Best Regards,
> Konstantin.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://knife.lugatgt.org/pipermail/mlpack/attachments/20170307/9c15f579/attachment-0001.html>


More information about the mlpack mailing list