\lyxframeend
                 {}\lyxframe{Outline}
The control sequence at the end of the top line
of your error message was never \def'ed. If you have
misspelled it (e.g., `\hobx'), type `I' and the correct
spelling (e.g., `I\hbox'). Otherwise just continue,
and I'll forget about whatever was undefined.

After comparing my document to example (working) beamer documents I’ve found out that you must have an EndFrame command after your last frame.Too bad it wasn’t documented anywhere I’ve found as this little thing drove me crazy.

For sometime before letting go of the issue, I’ve tried to correct the direction of the label. It could have been fixed by marking the float as English text in LyX (I use it for editing Hebrew papers, as it makes editing multi-lingual texts easy), effectively adding \selectlanguage{english} before the \begin{algorithm} line. However, it resulted in display mess in LyX and some LaTeX errors when compiling (also it seemed LaTeX could complete the first pass).

Today, I’ve decided to tackle and solve the problem for a new paper I’m writing. After searching a bit for a way to override the Algorithm environment label, I’ve found a general way to override a float label in LaTeX:

\floatname{floatname}{floatlabel}

So for our case it would look like this:

\floatname{algorithm}{אלגוריתם}

You should put it in your LaTeX preamble (Documents->Settings in LyX). This neatly fixes the problem (see screenshot below).

While LyX doesn’t support many of the great packages out there like mathtools (which I really hope it will someday), you can add some support to your documents. At the beginning of the document insert a comment, via Insert->Note->Comment. Inside the newly created comment insert a math-macro via Insert->Math->Macro. In the name part, put the name of the command you want to add support for. In the second box (caption LyX), use existing LyX commands to mimic how the macro will look like. For example, this is what it looks like for the \coloneqq macro (from the great mathtools package):

After adding the math macro in the comment, when you will use the macro inside formulas it will display nicely:

A little explanation how things work. When you define a math macro in LyX, LyX does two things:

1. Inserts LaTeX code to create the macro.
2. Displays the macro nicely when editing the document.

While the latter is desirable, the former is problematic. If LyX inserts LaTeX code to define the existing macro, it will cause errors. So when you put the LyX macro in the comment environment, the code LyX generates gets ignored and only the second, desirable, outcome is achieved.

So I wrote a patch for the he.po file that fixes the problem.

Index: he.po
===================================================================
--- he.po	(revision 28095)
+++ he.po	(working copy)
@@ -15050,15 +15050,15 @@
 
 #: src/Buffer.cpp:2902
 msgid "\\arabic{enumi}."
-msgstr "\\arabic{enumi}."
+msgstr ".\\arabic{enumi}"
 
 #: src/Buffer.cpp:2908
 msgid "\\roman{enumiii}."
-msgstr "\\roman{enumiii}."
+msgstr ".\\roman{enumiii}"
 
 #: src/Buffer.cpp:2911
 msgid "\\Alph{enumiv}."
-msgstr "\\Alph{enumiv}."
+msgstr ".\\Alph{enumiv}"
 
 #: src/Buffer.cpp:2928 src/insets/InsetCaption.cpp:293
 msgid "Senseless!!! "

While this patch is against the SVN trunk (LyX 2.0), it could also be applied manually to older LyX versions.
After yo apply the patch you should do the following (I assume that you are in the root of the LyX source tree):

cd po/
make he.gmo
cd ../
make install

Now restart LyX and you will see the numbering appear as it should.

LaTeX Error: Command \textquotedbl unavailable in encoding HE8

This could be solved by adding the following line to your LaTeX preamble:

\let\textquotedbl="

This line will define the \textquotedb to be the double quotation mark, and the document will compile look the same as it looked in older versions of LyX.