Academic literature on the topic 'Sleeping Bandits'

When confronted with massive data streams, summarizing data with dimension reduction methods such as PCA raises theoretical and algorithmic pitfalls. A principal curve acts as a nonlinear generalization of PCA, and the present paper proposes a novel algorithm to automatically and sequentially learn principal curves from data streams. We show that our procedure is supported by regret bounds with optimal sublinear remainder terms. A greedy local search implementation (called slpc, for sequential learning principal curves) that incorporates both sleeping experts and multi-armed bandit ingredients is presented, along with its regret computation and performance on synthetic and real-life data.

We compared 19 stocks of Trypanosoma brucei rhodesiense collected in 1991 and 1994 from Tanzania with representative stocks from other foci of Rhodesian sleeping sickness in Zambia, Kenya and Uganda. Stocks were characterized by isoenzyme electrophoresis, restriction fragment length polymorphisms in variant surface glycoprotein genes and random amplification of polymorphic DNA; the banding patterns obtained were coded for numerical analysis. In addition, the Tanzanian stocks were compared by pulsed field gel electrophoresis. Overall the Tanzanian stocks formed a homogeneous group and the predominant genotype isolated in 1991 was still present in the 1994 sample, although at a reduced level. The Tanzanian stocks were distinct from representative stocks from other East African foci. This observation does not support the proposal that there are northern and southern strains of T. b. rhodesiense, but is consistent with the view that T. b. rhodesiense stocks form a mosaic of different genotypes varying from focus to focus in East Africa.

Abstract Hurler syndrome (HS; mucopolysaccharidosis type I) is caused by severe mutations in the iduronidase (IDUA) gene, leading to multi-organ system dysfunction due to the toxic accumulation of glycosaminoglycans. Although allogeneic hematopoietic cell transplantation (HCT) has been shown to provide the IDUA protein and to reverse many of the manifestations of HS, allogeneic HCT is associated with significant morbidity and mortality. We hypothesized that an advantageous alternative strategy may be to induce gene-corrected autologous pluripotent cells to become hematopoietic stem cells, which then provide the missing IDUA enzyme. Because patient-specific embryonic stem cell isolation is not practical, recent strategies have been developed that reprogram adult cells to acquire pluripotency. Such induced pluripotent stem (iPS) cells can be created from fibroblasts or mesenchymal stromal cells (MSCs). As a first step in testing of iPS cells for gene-corrected HS treatment, we isolated host MSCs from the bone chips of a 9-year-old boy with HS who had undergone spinal surgery 8 years after successful allogeneic HCT. HS-MSCs expressed no IDUA, confirming a lack of contamination from either donor-derived hematopoietic cells or MSCs. To create HS-iPS cells, HS-MSCs were transduced with viral vectors carrying reprogramming transcription factors (OCT4, SOX2, KLF4, and c-MYC) that are typically associated with pluripotency and expressed at high levels in embryonic but not adult stem cells. Transduced cells were cultured on supportive stroma of irradiated mouse embryo fibroblasts. Within several weeks, colonies of iPS cells emerged from the two-dimensional culture. When compared to MSCs, the HS-iPS cells showed persistent mRNA expression of OCT3/4 and SOX2 and transient mRNA expression of c-MYC and KLF4, which is expected to occur in the wild-type iPS cells. HS-iPS cells expressed protein markers characteristic of reprogrammed immature cells: OCT3/4, NANOG, stage-specific embryonic antigens (SSEA) 3 and 4, tumor rejection antigens (TRA) 1–60 and 1–81, and alkaline phosphatase. HS-iPS cells had normal male karyotype as determined by chromosomal G-banding. As a second step in creating gene-corrected HS-iPS cells, we employed the non-viral Sleeping Beauty (SB) transposon system (because of the less random pattern of genome integration when compared to viral vectors). Human HS-iPS cells were co-nucleofected with an SB transposon that harbored the human IDUA gene and an expression cassette of the green fluorescent protein (GFP) along with an SB transposase plasmid that provides the enzymatic machinery necessary for integration into TA dinucleotide sites within the genome. Two weeks after nucleofection 10%-15% of HS-iPS cells expressed GFP. Total glycosaminoglycans (a hallmark of the biochemical defect in HS) in unsorted cultures were decreased to wild-type levels. IDUA expression in unsorted cultures was approximately 10% of wild-type IDUA levels, which is within the range sufficient for phenotypic rescue in HS patients after allogeneic HCT. Experiments are ongoing, and data will be presented in regards to: a) map transposon insertions in the genome to prove stable transgenesis by transposition; b) characterization of the differentiation potential of the corrected HS-iPS cells into various mesodermal lineages relevant to rescue of the clinical phenotype associated with HS (hematopoietic, chondrogenic, and osteogenic); c) assessment of development and consequences of cellular pathology in numerous tissue types affected by IDUA deficiency. To our knowledge these are the first data to report that autologous iPS cells can be obtained from HS patients. In summary, HS-iPS cells present an opportunity to use the hematopoietic progeny of gene-corrected autologous cells clinically in a manner that may preclude the immunologic complications of allogeneic transplantation.

Background: Remission of metabolic syndrome has been documented with gastric banding surgery. We document a clinical case of remission with lifestyle interventions. Hypothesis: A long term whole food plant based diet (WFBD) and lifestyle changes could potentially stop the metabolic syndrome. Methods: A 58 year old Caucasian from the US participated of a program in Weimar California. He had a history of diabetes mellitus type 2 for 5 years with peripheral neuropathy for 1.5 years, he took metformin 500 mg bid. He was also hypertensive, he used losartan 50 mg bid and furosemide 40 mg qd. The medical lifestyle program included a WFPBD, exercise, two meals a day, sleeping hygiene, cooking classes together with psychological, medical and spiritual consultations. The program lasted 18 days. The patient continued with the program at home. Laboratory values are reported in mg/dL Results: At baseline blood pressure was 158/78 mm hg, he claim to be compliant with medication, fasting blood glucose (FBG) was 230 and weighed 381.4 pounds. At the end of the residential 18 days he had lost 16.2 pounds, blood pressure was 116/72 mm hg, FBG of 110. The patient continued with the learned lifestyle at home. 142 days later he is seen by a physician who stops all medication, he had lost 83.4 pounds, blood pressure was 116/68 mm hg, FBG was 89 mg/dl, hemoglobin A1c was 5.8%. In 286 days he is seen again by a physician, he is now 270 pounds, the patient continues with no medication. 326 days after the start of the program he does lab work, he had a total cholesterol of 140, triglycerides of 56, HDL of 68, LDL of 61 and hemoglobin A1c of 5.5%. 470 days after the program a physician recorded that he had lost 139.4 pounds from the start of the program. He is not using any medication. 524 days after the program he now weighs 240 pounds, having lost 141.4 pounds. 917 days after the end of the program labs are taken: cholesterol 139, triglycerides 55, HDL 73, LDL 55, FBG 99, hemoglobin A1C 5.3%, CHL/HDL ratio of 1.9, blood pressure of 118/70 mm hg. The patient reports feeling well. Conclusions: A change in lifestyle that includes a plant based diet was an effective technique to stop the metabolic syndrome without the need of surgery.

The elicitation and aggregation of preferences is often the key to making better decisions. Be it a perfume company wanting to relaunch their 5 most popular fragrances, a movie recommender system trying to rank the most favoured movies, or a pharmaceutical company testing the relative efficacies of a set of drugs, learning from preference feedback is a widely applicable problem to solve. One can model the sequential version of this problem using the classical multiarmed-bandit (MAB) (e.g., Auer, 2002) by representing each decision choice as one bandit-arm, or more appropriately as a Dueling-Bandit (DB) problem (Yue \& Joachims, 2009). Although DB is similar to MAB in that it is an online decision making framework, DB is different in that it specifically models learning from pairwise preferences. In practice, it is often much easier to elicit information, especially when humans are in the loop, through relative preferences: `Item A is better than item B' is easier to elicit than its absolute counterpart: `Item A is worth 7 and B is worth 4'. However, instead of pairwise preferences, a more general $k$-subset-wise preference model $(k \ge 2)$ is more relevant in various practical scenarios, e.g. recommender systems, search engines, crowd-sourcing, e-learning platforms, design of surveys, ranking in multiplayer games. Subset-wise preference elicitation is not only more budget friendly, but also flexible in conveying several types of feedback. For example, with subset-wise preferences, the learner could elicit the best item, a partial preference of the top 5 items, or even an entire rank ordering of a subset of items, whereas all these boil down to the same feedback over pairs (subsets of size 2). The problem of how to learn adaptively with subset-wise preferences, however, remains largely unexplored; this is primarily due to the computational burden of maintaining a combinatorially large, $O(n^k)$, size of preference information in general (for a decision problem with $n$ items and subsetsize $k$). We take a step in the above direction by proposing ``Battling Bandits (BB)''---a new online learning framework to learn a set of optimal ('good') items by sequentially, and adaptively, querying subsets of items of size up to $k$ ($k\ge 2$). The preference feedback from a subset is assumed to arise from an underlying parametric discrete choice model, such as the well-known Plackett-Luce model, or more generally any random utility (RUM) based model. It is this structure that we leverage to design efficient algorithms for various problems of interest, e.g. identifying the best item, set of top-k items, full ranking etc., for both in PAC and regret minimization setting. We propose computationally efficient and (near-) optimal algorithms for above objectives along with matching lower bound guarantees. Interestingly this leads us to finding answers to some basic questions about the value of subset-wise preferences: Does playing a general $k$-set really help in faster information aggregation, i.e. is there a tradeoff between subsetsize-$k$ vs the learning rate? Under what type of feedback models? How do the performance limits (performance lower bounds) vary over different combinations of feedback and choice models? And above all, what more can we achieve through BB where DB fails? We proceed to analyse the BB problem in the contextual scenario – this is relevant in settings where items have known attributes, and allows for potentially infinite decision spaces. This is more general and of practical interest than the finite-arm case, but, naturally, on the other hand more challenging. Moreover, none of the existing online learning algorithms extend straightforwardly to the continuous case, even for the most simple Dueling Bandit setup (i.e. when $k=2$). Towards this, we formulate the problem of ``Contextual Battling Bandits (C-BB)'' under utility based subsetwise-preference feedback, and design provably optimal algorithms for the regret minimization problem. Our regret bounds are also accompanied by matching lower bound guarantees showing optimality of our proposed methods. All our theoretical guarantees are corroborated with empirical evaluations. Lastly, it goes without saying, that there are still many open threads to explore based on BB. These include studying different choice-feedback model combinations, performance objectives, or even extending BB to other useful frameworks like assortment selection, revenue maximization, budget-constrained bandits etc. Towards the end we will also discuss some interesting combinations of the BB framework with other, well-known, problems, e.g. Sleeping / Rotting Bandits, Preference based Reinforcement Learning, Learning on Graphs, Preferential Bandit-Convex-Optimization etc.

The multi-armed bandit (MAB) problem provides a convenient abstraction for many online decision problems arising in modern applications including Internet display advertising, crowdsourcing, online procurement, smart grids, etc. Several variants of the MAB problem have been proposed to extend the basic model to a variety of practical and general settings. The sleeping multi-armed bandit (SMAB) problem is one such variant where the set of available arms varies with time. This study is focused on analyzing the efficacy of the Thompson Sampling algorithm for solving the SMAB problem. Any algorithm for the classical MAB problem is expected to choose one of K available arms (actions) in each of T consecutive rounds. Each choice of an arm generates a stochastic reward from an unknown but fixed distribution. The goal of the algorithm is to maximize the expected sum of rewards over the T rounds (or equivalently minimize the expected total regret), relative to the best fixed action in hindsight. In many real-world settings, however, not all arms may be available in any given round. For example, in Internet display advertising, some advertisers might choose to stay away from the auction due to budget constraints; in crowdsourcing, some workers may not be available at a given time due to timezone difference, etc. Such situations give rise to the sleeping MAB abstraction. In the literature, several upper confidence bound (UCB)-based approaches have been proposed and investigated for the SMAB problem. Our contribution is to investigate the efficacy of a Thomp-son Sampling-based approach. Our key finding is to establish a logarithmic regret bound, which non-trivially generalizes a similar bound known for this approach in the classical MAB setting. Our bound also matches (up to constants) the best-known lower bound for the SMAB problem. Furthermore, we show via detailed simulations, that the Thompson Sampling approach in fact outperforms the known algorithms for the SMAB problem.

The multi-armed bandit (MAB) problem provides a convenient abstraction for many online decision problems arising in modern applications including Internet display advertising, crowdsourcing, online procurement, smart grids, etc. Several variants of the MAB problem have been proposed to extend the basic model to a variety of practical and general settings. The sleeping multi-armed bandit (SMAB) problem is one such variant where the set of available arms varies with time. This study is focused on analyzing the efficacy of the Thompson Sampling algorithm for solving the SMAB problem. Any algorithm for the classical MAB problem is expected to choose one of K available arms (actions) in each of T consecutive rounds. Each choice of an arm generates a stochastic reward from an unknown but fixed distribution. The goal of the algorithm is to maximize the expected sum of rewards over the T rounds (or equivalently minimize the expected total regret), relative to the best fixed action in hindsight. In many real-world settings, however, not all arms may be available in any given round. For example, in Internet display advertising, some advertisers might choose to stay away from the auction due to budget constraints; in crowdsourcing, some workers may not be available at a given time due to timezone difference, etc. Such situations give rise to the sleeping MAB abstraction. In the literature, several upper confidence bound (UCB)-based approaches have been proposed and investigated for the SMAB problem. Our contribution is to investigate the efficacy of a Thomp-son Sampling-based approach. Our key finding is to establish a logarithmic regret bound, which non-trivially generalizes a similar bound known for this approach in the classical MAB setting. Our bound also matches (up to constants) the best-known lower bound for the SMAB problem. Furthermore, we show via detailed simulations, that the Thompson Sampling approach in fact outperforms the known algorithms for the SMAB problem.

Siberia is the Sleeping Land, a huge subcontinent barely awakened by the first nomads who arrived here uncounted millennia ago, and still, toward the end of the nineteenth century, so devoid of people that in much of it you could travel hundreds of miles in almost any direction and see no evidence that humans had ever existed. Yet the aim of Tsar Alexander III in committing Russia to carve the longest railroad in the world across this great nowhere was not primarily to provide an avenue for settlement by immigrants from overcrowded and often impoverished European Russia. A hundred years after the railroad’s completion, Siberia remains today one of the least-populated places on earth. No, the primary purpose was empire building. It was a way to gain better access to the region’s dazzling natural riches, which were the property of the tsar; to protect the eastern flank of the empire against Chinese and Japanese designs and provide a launching pad for Russia’s own designs to the east; and to bind together a string of Russian settlements flung out over a contiguous land mass larger than that ever claimed by any other single entity. To accomplish its goal of uniting Vladivostok on the Pacific with Moscow and then St. Petersburg on the Atlantic, Russia had to do something that nearly all engineers at the time judged impossible—carve a passable corridor through a continent’s worth of forest, bog, permafrost, stone, and swamp. The work was done by free peasants, imported labor, and prisoners wielding wooden shovels, specially designed machinery, dynamite to blast through permafrost, and bonfires to melt it. Workers had to contend with plague and cholera, searing arctic winters and blistering summers, and attacks by insects, tigers, and bandits. It took twenty-five years from the first felled tree to the last spike, it cost roughly a billion rubles all told, or perhaps as much as seven billion in today’s dollars, and by the time it was done in 1916, the empire was nearly bankrupt and on the verge of collapse. But to a large extent, the effort to bind together at least Russia itself, if not the larger empire, succeeded.